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Preface 


The  RAND  Corporation  undertook  this  analysis  to  assist  the  National  Institute  for 
Occupational  Safety  and  Health  (NIOSH)  in  developing  and  applying  a  framework 
for  conducting  external  reviews  of  the  impact  of  its  research  activities.  NIOSH  initi¬ 
ated  the  reviews  to  better  ensure  the  applicability  of  its  work  in  preventing  work-related 
injuries  and  illnesses.  The  analysis  is  also  intended  to  provide  assistance  to  other  federal 
research  agencies  facing  similar  concerns. 

This  book  reports  on  phase  II  of  the  project,  “Analytical  and  Operational  Support 
for  NIOSH  External  Program  Review.”  In  particular,  it  describes  the  methodology  we 
developed  to  assist  NIOSH  programs  in  preparing  for  external  review  by  the  National 
Academies.  It  also  details  the  use  of  logic  models,  outcome  worksheets,  and  outcome 
narratives  as  key  tools  in  preparing  evidence  packages  that  demonstrate  and  commu¬ 
nicate  the  impact  NIOSH  research  activities  have  had  in  contributing  to  reduction  in 
hazardous  exposures,  occupational  illnesses,  injuries,  and  fatalities.  This  book  should 
be  of  interest  to  mission-oriented  federal  research  and  development  (R&D)  agencies 
that  are  either  planning  for  or  undergoing  external  review;  the  program  managers  and 
research  scientists  within  these  agencies;  program  evaluators  (public  and  private);  non- 
R&D  federal  agencies  interested  in  evaluating  the  outcomes  of  research  programs;  sci¬ 
ence  policymakers;  and  research  funders. 

Other  RAND  publications  that  might  interest  the  reader  include  the  following: 

•  The  Returns  from  Arthritis  Research,  Vol.  1:  Approach,  Analysis  and  Recommenda¬ 
tions  (Wooding  et  ah,  2004) 

•  Using  Logic  Models  for  Strategic  Planning  and  Evaluation:  Application  to  the 
National  Center  for  Injury  Prevention  and  Control  (Greenfield,  Williams,  and 
Eiseman,  2006). 
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The  RAND  Safety  and  Justice  Program 

This  research  was  conducted  under  the  auspices  of  the  Safety  and  Justice  Program 
within  RAND  Infrastructure,  Safety,  and  Environment  (ISE).  The  mission  of  ISE  is 
to  improve  the  development,  operation,  use,  and  protection  of  society’s  essential  physi¬ 
cal  assets  and  natural  resources  and  to  enhance  the  related  social  assets  of  safety  and 
security  of  individuals  in  transit  and  in  their  workplaces  and  communities.  Safety 
and  Justice  Program  research  addresses  occupational  safety,  transportation  safety,  food 
safety,  and  public  safety — including  violence,  policing,  corrections,  substance  abuse, 
and  public  integrity. 

Questions  or  comments  about  this  book  should  be  sent  to  the  project  leader, 
Valerie  L.  Williams  (Valerie_Williams@rand.org).  Information  about  the  Safety  and 
Justice  Program  is  available  online  (http://www.rand.org/ise/safety).  Inquiries  about 
research  projects  should  be  sent  to  the  following  address: 

Greg  Ridgeway,  Acting  Director 
Safety  and  Justice  Program,  ISE 
RAND  Corporation 
P.O.  Box  2138 
1776  Main  Street 
Santa  Monica,  CA  90407-2138 
310-393-0411,  x7734 
Greg_Ridgeway@rand.org 
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Summary 


The  evaluation  of  research  impact  is  a  topic  of  enduring  interest  to  research  funders  and 
performers  of  research.  Research  impact  refers  to  the  contribution  of  research  activities  to 
desired  societal  outcomes,  such  as  improved  health,  environment,  economic,  and  social 
conditions.  In  recent  years,  this  interest  has  grown  because  of  governments’  desire  to 
understand  the  impact  of  publicly  funded  research  for  the  purpose  of  budgeting  and 
resource  allocation  decisions,  both  nationally  and  internationally.  In  the  United  States, 
the  1993  Government  Performance  and  Results  Act  (GPRA)  (Pub.  L.  No.  103-62)  and 
the  2002  Program  Assessment  Rating  Tool  (PART)  are  the  most  recent  manifestations 
of  the  public’s  concern  about  the  payoff  of  federally  funded  research.  These  policies, 
which  require  all  federal  programs  to  conduct  assessments  of  their  own  performance, 
present  special  challenges  for  research  programs  because  of  the  methodological  diffi¬ 
culty  of  measuring  the  impact  of  research. 

The  difficulties  associated  with  tracking  and  measuring  the  societal  outcomes 
of  research  has  caused  this  area  of  evaluation  to  lag  other  types  of  evaluation  that 
seek  to  assess  other  dimensions  of  research,  such  as  quality,  relevance,  and  productiv¬ 
ity.  Despite  these  difficulties,  approaches  to  evaluating  the  impact  of  research  have 
progressed  substantially  in  the  past  decade.  Technometrics,  sociometrics,  bibliomet- 
rics,  value-mapping,  expert  review,  and  case  studies  represent  both  quantitative  and 
qualitative  means  of  assessing  the  benefits  of  research  to  industry,  government,  and  the 
public. 

Federal  agencies  often  employ  multiple  types  of  expert  review  to  evaluate  research 
impact.  In  the  past  few  decades,  use  of  expert  panels  has  become  commonplace  for 
evaluating  larger  units,  such  as  research  groups,  institutes,  and  research  programs. 
In  addition  to  evaluating  scientific  merit,  these  panels  often  assess  the  socioeconomic 
impact  of  research. 


Expert  Review  of  NIOSH  Programs  by  the  National  Academies 

In  September  2004,  the  National  Institute  for  Occupational  Safety  and  Health 
(NIOSH)  contracted  with  the  National  Academies  to  conduct  external  reviews  of  up 
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to  15  of  its  research  programs.  The  purpose  of  the  reviews  was  to  judge  the  extent 
to  which  each  program’s  research  was  relevant  to  real-world  occupational  safety  and 
health  (OSH)  problems;  contributed  to  reductions  in  occupational  hazardous  expo¬ 
sures,  illnesses,  and  injuries;  and  was  effective  in  targeting  new  research  areas  and 
identifying  emerging  issues. 

Eight  research  programs  were  reviewed  by  individual  evaluation  committees 
(ECs)  composed  of  persons  with  expertise  appropriate  to  evaluating  the  specific  pro¬ 
gram.  In  many  cases,  experts  were  recruited  from  stakeholder  groups  (such  as  labor 
unions  and  industry).  Experts  in  technology  transfer  and  program  evaluation  were 
also  included.  In  conducting  their  evaluations,  the  ECs  ascertained  whether  NIOSH  is 
doing  the  right  things  (relevance)  and  whether  those  things  are  improving  health  and 
safety  in  the  workplace  (impact). 

To  maintain  consistency  in  the  evaluation  across  the  independent  ECs,  the 
National  Academies  appointed  a  committee  of  14  members,  including  persons  with 
expertise  in  occupational  medicine  and  health,  industrial  health  and  safety,  industrial 
hygiene,  epidemiology,  civil  and  mining  engineering,  sociology,  program  evaluation, 
communication,  and  toxicology;  representatives  of  industry  and  of  the  workforce;  and 
a  scientist  experienced  in  international  occupational-health  issues.  This  committee, 
referred  to  as  the  Framework  Committee,  developed  a  guide  to  provide  a  common 
structure  for  the  review  of  the  different  research  programs.  The  guide,  called  the  frame¬ 
work  document,  outlines  the  evaluation  criteria  to  be  used  by  the  ECs,  the  information 
needs,  and  the  specific  evaluation  questions  to  be  considered. 

The  framework  document  recommended  that  the  ECs  consider  the  available  evi¬ 
dence  of  reduced  work-related  risks  and  adverse  effects  (hazardous  exposures,  illnesses, 
fatalities,  and  injuries)  and  external  factors  related  to  the  changes.  A  Ending  of  high 
impact  required  the  EC’s  judgment  that  the  research  program  had  contributed  to  these 
end  outcomes.  So,  for  example,  high  impact  could  mean  that  outcomes  had  occurred 
earlier  than  they  otherwise  would  have  or  were  better  than  they  would  have  been  in 
the  absence  of  the  research  program.  A  Ending  of  high  impact  could  also  result  if 
external  factors  beyond  NIOSH’s  control  had  impeded  achievement  of  end  outcomes. 
The  criteria  for  assessing  relevance  centered  on  whether  the  program  appropriately  set 
priorities  among  research  needs  and  the  assessment  of  how  engaged  the  program  is  in 
appropriate  transfer  activities. 

NIOSH  asked  RAND  to  assist  it  in  preparing  for  and  engaging  in  external  pro¬ 
gram  reviews.  Our  activities  took  place  in  two  phases:  research  and  design  (phase  I) 
and  implementation  (phase  II).  In  phase  I,  we  met  with  other  federal  agencies  to  gain 
perspective  about  different  approaches  to  external  review;  developed  a  set  of  guidance 
principles  to  assist  NIOSH  programs  throughout  the  external  review  process;  and  cre¬ 
ated  a  detailed  set  of  specifications  for  preparing  a  model  package  of  information,  or 
evidence  packages ,  to  give  to  the  reviewers  (Greenfield,  Balakrishnan,  et  ah,  2006). 
In  phase  II,  we  worked  with  selected  NIOSH  programs  undergoing  external  review, 
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which  included  assistance  with  preparing  evidence  packages.  Phase  II  efforts  led  to  the 
development  of  a  set  of  tools  that  were  instrumental  to  demonstrating  and  communi¬ 
cating  the  impact  of  research  activities  on  achieving  outcomes. 

This  document  reports  on  phase  II  of  the  project.  In  particular,  it  describes  the 
tools  developed — logic  models,  outcome  worksheets,  and  outcome  narratives — and 
examines  their  use  in  preparing  evidence  packages. 


Demonstrating  Impact 

Logic  models  and  outcome  worksheets  were  used  primarily  to  help  NIOSH  programs 
demonstrate  how  their  research  was  intended  to  achieve  program  outcomes.  Each  is 
discussed  in  more  detail  in  this  section. 

Logic  Models 

A  central  challenge  in  demonstrating  the  impact  of  research  programs  is  describing 
the  path  by  which  research  achieves  its  intended  outcomes  (in  NIOSH’s  case,  reduc¬ 
tions  in  work-related  hazardous  exposures,  illnesses,  or  injuries).  Logic  models  can 
help  with  this  demonstration.  A  logic  model  is  a  visual  depiction  of  the  stages  across 
which  research  inputs  are  translated  into  outcomes.  Such  depictions  can  help  reviewers 
understand  how  research  activities  achieve  societal  objectives  or  impacts.  Logic  models 
provide  a  comprehensive  view  of  a  research  program:  what  it  does,  whom  it  affects, 
and  the  expected  outcomes  that  can  form  the  basis  of  the  evaluation.  Logic  models 
can  also  define  the  domain  of  analysis  for  evaluating  impact.  By  showing  the  multiple 
contributors  to  any  given  end  outcome,  the  logic  model  helps  define  the  program’s 
sphere  of  influence — i.e.,  for  what  the  program  can  take  credit  and  for  what  it  can  be 
held  responsible. 

There  are  several  ways  to  develop  and  customize  logic  models  for  specific  research 
programs.  Because  a  logic  model  is  an  abstraction  that  omits  detail  for  the  sake  of 
clarity  of  representation,  the  trade-off  between  detailed  accuracy  and  clarity  of  pre¬ 
sentation  varies  according  to  context  and  purpose.  In  developing  logic  models  with 
NIOSH  programs,  clearly  depicting  the  paths  to  outcomes  was  the  priority.  Thus,  we 
maintained  the  linear  flow  of  the  logic  model,  using  feedback  loops  and  other  divergent 
pathways  only  in  rare  cases.  Moreover,  these  logic  models  were  accompanied  by  text  in 
the  evidence  packages,  and  many  of  the  complex  details  that  were  omitted  in  the  logic 
model  were  explained  in  these  narratives. 

Outcome  Worksheets 

Logic  models  were  used  to  assist  NIOSH  programs  in  presenting  an  overarching  view 
of  their  programs  and,  in  so  doing,  articulate  their  program  theory.  However,  the 
evidence  of  impact  needed  for  the  evidence  packages  required  descriptions  of  cases  in 
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which  NIOSH  programs  achieved  specific  outcomes.  The  guidance  we  provided  to 
NIOSH  research  programs  for  describing  their  path  to  outcomes  is  based  on  the  his¬ 
torical  tracing  method.  This  method  traces  a  series  of  interrelated  events  either  going 
forward  from  the  research  activities  to  downstream  outcomes  or  working  backward 
from  an  outcome  to  precursor  research.  A  tool  that  supported  the  use  of  this  method 
was  the  outcome  worksheet.  An  outcome  worksheet  is  a  spreadsheet  that  details  how 
specific  outcomes  were  achieved,  based  on  the  path  described  by  the  logic  model. 
Figure  S.l  illustrates  the  creation  of  the  outcome  worksheets  from  the  logic  model. 

As  shown  in  Figure  S.l,  the  logic  model  begins  with  the  “Inputs”  box  in  red  and 
ends  with  the  “End  outcomes”  box  in  purple.  The  outcome  worksheet  (shown  in  the 
lower  half  of  Figure  S.l)  includes  the  strategic  and  intermediate  goals  and  reverses  the 
order  of  the  logic  model  elements,  with  the  end  outcomes  at  the  far  left  end  and  the 
inputs  at  the  right.  The  lines  within  each  of  the  colored  boxes  represent  the  text  that  is 
typically  included  in  the  logic  models. 

We  developed  the  outcome  worksheet  to  assist  NIOSH  researchers  in  think¬ 
ing  through  the  causal  linkages  between  specific  outcomes  and  research  activities, 

Figure  S.l 

Moving  from  the  Logic  Model  to  the  Outcome  Worksheet 
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determining  the  data  needed  to  provide  evidence  of  impact,  and  structuring  the  evi¬ 
dence  in  a  systematic  framework. 

The  first  step  in  creating  an  outcome  worksheet  involves  deciding  whether  to 
trace  the  research  path  forward  (i.e.,  from  research  activities  to  outcomes)  or  backward. 
Forward  tracing  can  capture  a  comprehensive  view  of  a  research  project’s  or  program’s 
effects.  Because  the  path  leads  from  the  research,  the  connection  to  the  research  is 
ensured.  In  contrast,  backward  tracing  usually  focuses  on  a  single  outcome  of  impor¬ 
tance  and  follows  the  trail  back  through  those  developments  that  were  critical  in  reach¬ 
ing  the  identified  outcome.  One  implication  of  backward  tracing  is  that  it  highlights 
activities  that  led  to  anticipated  outcomes  and  may  not  capture  the  broader  range  of 
outcomes  to  which  forward  tracing  may  lead  or  may  select  only  the  most-positive  cases 
of  outcomes.  In  NIOSH’s  case,  the  National  Academies’  review  focused  on  impact, 
and,  as  shown  in  Figure  S.l,  outcomes  were  the  natural  starting  point,  followed  by  cus¬ 
tomers,  transfer  activities,  outputs,  research  activities,  and,  finally,  inputs.  Not  only  did 
this  backward  tracing  reinforce  the  emphasis  on  outcomes,  it  also  oriented  researchers 
toward  a  collective  body  of  research  rather  than  on  individual  research  projects. 

In  addition  to  identifying  and  structuring  information,  the  format  of  the  out¬ 
come  worksheet  also  enabled  quick  review  and  analysis  of  a  large  amount  of  informa¬ 
tion.  Finally,  the  outcome  worksheet  was  critical  in  helping  research  programs  prepare 
outcome  narratives,  our  final  tool,  for  communicating  impact. 


Communicating  Impact 

The  logic  models  and  the  information  from  the  outcome  worksheets  became  part  of 
a  larger  set  of  materials  assembled  in  the  form  of  an  evidence  package  and  submitted 
to  the  reviewers.  A  key  component  of  the  package  was  the  outcome  narrative,  which 
helped  to  communicate  the  impact  of  NIOSFI  programs  by  describing  how  specific 
research  activities  contributed  to  intermediate  or  end  outcomes.  The  outcome  narrative 
served  as  our  primary  tool  in  communicating  impact  and  is  described  in  more  detail 
in  this  section. 

Outcome  Narratives 

The  central  purpose  of  the  evidence  package  is  to  communicate  to  reviewers  how 
research  activities  have  contributed  to  societal  outcomes.  The  reviewers  were  expected 
to  use  their  expert  judgment  and  knowledge  of  the  field  to  evaluate  the  claims  in  the 
evidence  package  about  the  role  of  NIOSFI  programs  in  achieving  intermediate  out¬ 
comes  (such  as  changes  in  workplace  practices)  or  end  outcomes  (such  as  reductions 
in  hazardous  exposures).  These  claims  of  impact  were  presented  in  the  form  of  out¬ 
come  narratives  that  described  specific  instances  of  research  that  led  to  outcomes.  To 
structure  the  outcome  narrative,  we  again  drew  on  the  elements  of  the  logic  model. 
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Figure  S.2  illustrates  the  connections  among  the  tools  we  developed,  including  the 
outcome  narrative. 

The  outcome  narrative  had  five  major  sections,  accompanied  by  specific 
questions. 

•  Issue:  What  is  the  major  societal  problem  ?  The  narrative  should  begin  by  defining 
the  issue  and  its  significance,  why  it  exists,  and  who  is  affected. 

•  Approach :  What  approach  has  been  used  to  address  this  issue ?  This  section  describes 
the  research  strategies  that  have  been  used  to  address  the  issue. 

•  Outputs  and  Transfer:  What  were  the  major  outputs  from  this  research  areal  How 
and  to  whom  were  the  products  transferred ?  This  section  highlights  the  relevance  of 
outputs  and  transfer  activities. 

•  Intermediate  Outcomes  (or  End  Outcomes):  What  effect  did  the  outputs  have  on 
the  broader  community ?  This  section  emphasizes  the  effect  of  program  outputs 
and  establishes  a  causal  thread  by  describing  intermediate  and  final  customers’ 
responses  to  program  outputs.  The  discussion  plausibly  links  to  some  output. 

•  What  Is  Ahead:  What  are  some  specific  research  activities  currently  under  way  or 
in  planning  in  response  to  the  problem?  This  is  an  optional  section  that  showcases 
activities  or  outputs  currently  in  progress  or  in  the  planning  stages.  It  should 
include  work  that  has  not  yet  achieved  intermediate  outcomes  status  but  that  is 
clearly  on  the  horizon  (three  to  five  years  out). 


Other  Applications  of  These  Tools 

The  tools  described  in  this  book  have  uses  beyond  supporting  preparations  for  external 
review.  They  can  also  be  used  by  research  programs  to  conduct  outcome  monitoring, 
which  can  assist  program  managers  in  thinking  through  the  data  they  will  need  to 
demonstrate  program  effectiveness. 

Logic  models  can  also  support  project  planning  and  management,  as  they  provide 
a  structure  for  determining  whether  existing  strategic,  intermediate,  and  annual  goals 
are  aligned  with  program  operations.  These  goals  can  drive  the  development  of  mea¬ 
sures  that  can  gauge  the  progress  of  the  program  toward  achieving  outcomes. 

The  outcome  worksheets  are  useful  for  determining  the  appropriate  data  required 
for  outcome  monitoring  and  tracking.  Using  these,  research  programs  can  identify 
which  set  of  research  activities  have  been  linked  to  outcomes,  assess  the  extent  to  which 
transfer  activities  have  led  to  intended  customer  outcomes,  and  identify  the  range  of 
intended  and  unintended  outcomes  from  their  outputs.  Over  time,  these  worksheets 
can  become  the  foundation  of  a  database  that  tracks  uptake,  adoption,  and  utility  of 
research  outputs  by  different  customers.  This  could  enable  better  strategic  planning  of 
transfer  activities  and  working  more  effectively  with  partners  at  early  project  phases. 
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Figure  S.2 

Relationship  Among  the  Three  Tools 
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Finally,  the  outcome  narratives  are  useful  tools  for  communicating  impact  to 
audiences  beyond  reviewers.  An  effective  outcome  narrative  can  convey  the  value  of 
research  to  key  stakeholders,  who  often  prefer  reading  documents  that  get  to  the  point 
quickly  and  clearly.  The  concise  format  and  readable  layout  of  the  outcome  narrative 
ensures  that  the  investments  to  demonstrate  impact  can  be  accessed  and  appreciated 
by  a  broader  community. 


Acknowledgments 


A  number  of  individuals  contributed  to  both  this  project  and  the  writing  of  this  book. 
The  authors  wish  to  acknowledge  Victoria  Greenfield  for  her  leadership  and  vision.  Her 
work  in  the  initial  phase  of  this  project  laid  the  foundation  for  the  work  described  in 
this  book.  The  authors  wish  to  thank  Ray  Sinclair,  the  NIOSH  technical  monitor  of 
this  project,  for  his  support  and  enthusiasm  for  the  project.  His  tireless  efforts  through¬ 
out  the  project  ensured  successful  coordination  with  each  of  the  NIOSH  research 
programs  that  underwent  review  and  contributed  to  this  project’s  overall  success.  We 
thank  the  staff  of  the  Office  of  the  Director  of  NIOSH  for  the  opportunity  to  provide 
consultative  advice  about  the  design  of  the  framework  for  the  reviews  to  both  NIOSH 
and  the  National  Academies.  That  early  work  set  the  standards,  methods,  and  tone  for 
the  reviews  that  made  the  work  with  the  individual  programs  that  is  reflected  in  this 
book  possible. 

We  also  thank  the  NIOSH  directors,  deputy  directors,  program  managers,  coor¬ 
dinators,  researchers,  and  staff  who  worked  closely  with  us  throughout  the  preparation 
of  each  of  the  programs  for  external  review:  Gregory  Lotz  and  William  Murphy  (Hear¬ 
ing  Loss  Prevention);  Jeffrey  Kohler,  Giiner  Gurtunca,  and  Gerry  Finfinger  (Mining); 
David  Weissman  and  Ainsley  Weston  (Respiratory  Diseases);  Nancy  Stout  and  Her¬ 
bert  Linn  (Traumatic  Injury);  Frank  Hearl  and  Matt  Gillen  (Construction);  Teresa 
Schnorr  and  Allison  Tepper  (Health  Hazard  Evaluation);  and  Leslie  Boord,  Maryann 
D’Alessandro,  and  Roland  Berry  Ann  (Personal  Protective  Technology). 

We  give  special  thanks  to  Steven  Wooding  and  Susan  Cozzens,  who  served  as 
reviewers  for  this  book.  Their  comments  and  recommendations  strengthened  the  book 
considerably.  Any  remaining  errors  of  fact  or  interpretation  are  the  sole  responsibility 
of  the  authors. 


XIX 


Abbreviations 


AFL-CIO 

American  Federation  of  Labor  and  Congress  of  Industrial 
Organizations 

ANSI 

American  National  Standards  Institute 

ARC 

agricultural  research  center 

ASTM 

ASTM  International  (formerly,  the  American  Society  for  Testing  and 
Materials) 

BLS 

Bureau  of  Labor  Statistics 

CAD 

computer-aided  design 

CDC 

Centers  for  Disease  Control  and  Prevention 

CFOI 

Census  of  Fatal  Occupational  Injuries 

COPPE 

Committee  on  Personal  Protective  Equipment 

CPSC 

Consumer  Product  Safety  Commission 

CPWR 

Center  for  Construction  Research  and  Training,  formerly  known  as 
the  Center  to  Protect  Workers’  Rights 

CSC 

Construction  Steering  Committee 

DELS 

Division  of  Earth  and  Life  Studies 

DHHS 

U.S.  Department  of  Health  and  Human  Services 

DHS 

U.S.  Department  of  Homeland  Security 

DoD 

U.S.  Department  of  Defense 

DOI 

U.S.  Department  of  the  Interior 

DOL 

U.S.  Department  of  Labor 

XXI 


xxii  Demonstrating  and  Communicating  Research  Impact 


EC 

evaluation  committee 

EPA 

U.S.  Environmental  Protection  Agency 

ERC 

education  and  research  center 

EU 

European  Union 

FACE 

Fatality  Assessment  and  Control  Evaluation 

FSB 

Field  Studies  Branch 

FY 

fiscal  year 

GPRA 

Government  Performance  and  Results  Act 

HETAB 

Hazard  Evaluations  and  Technical  Assistance  Branch 

HEW 

U.S.  Department  of  Health,  Education,  and  Welfare 

HHE 

Health  Hazard  Evaluation 

IMIS 

Integrated  Management  Information  System 

IOM 

Institute  of  Medicine 

ISO 

International  Organization  for  Standardization 

LIUNA 

Laborers’  International  Union  of  North  America 

MESA 

Mine  Enforcement  and  Safety  Administration 

MMWR 

Morbidity  and  Mortality  Weekly  Report 

MRI 

magnetic  resonance  imaging 

MSHA 

Mine  Safety  and  Health  Administration 

NFPA 

National  Fire  Protection  Association 

NGO 

nongovernmental  organization 

NIH 

National  Institutes  of  Health 

NIOSH 

National  Institute  for  Occupational  Safety  and  Health 

NIST 

National  Institute  of  Standards  and  Technology 

NORA 

National  Occupational  Research  Agenda 

NSF 

National  Science  Foundation 

OMB 

Office  of  Management  and  Budget 

Abbreviations 


xxiii 


OSH 

occupational  safety  and  health 

OSHA 

Occupational  Safety  and  Health  Administration 

OSHAct 

Occupational  Safety  and  Health  Act  of  1970 

OSTP 

U.S.  Office  of  Science  and  Technology  Policy 

PART 

Program  Assessment  Rating  Tool 

PortMan 

portfolio  management 

PPT 

personal  protective  technology 

r2p 

research  to  practice 

R&D 

research  and  development 

RCT 

randomized  controlled  trial 

RDRP 

Respiratory  Diseases  Research  Program 

RIM 

reaction  injection  molding 

RQF 

research  quality  framework 

RVM 

research-value  mapping 

SEI 

Software  Engineering  Institute 

TOP 

targeted  outcomes  of  programs 

TRACES 

Technology  in  Retrospect  and  Critical  Events  in  Science 

UBC 

United  Brotherhood  of  Carpenters  and  Joiners  of  America 

UL 

Underwriters  Laboratories 

USDA 

U.S.  Department  of  Agriculture 

CHAPTER  ONE 


Introduction 


Evaluating  the  Impact  of  Research  Programs 

Evaluating  the  impact — and,  by  extension,  the  benefits — of  research  to  society  has  his¬ 
torically  been  of  great  interest  to  research  funders,  program  managers,  policymakers, 
researchers,  policy  analysts,  and  the  public.  However,  given  the  methodological  diffi¬ 
culties  associated  with  tracking  and  measuring  the  societal  outcomes  of  research,  this 
area  has  lagged  other  types  of  evaluation  that  seek  to  assess  other  aspects  of  research, 
such  as  quality,  relevance,  and  productivity.  Some  of  the  factors  that  complicate  the 
evaluation  of  research  impact  include  the  following: 

•  the  long  time  lag  between  the  completion  of  a  body  of  research  and  the  achieve¬ 
ment  of  its  full  impact,  especially  for  basic  research.  It  may  take  decades  or  more 
before  research  products  are  widely  adopted  and  produce  widespread  benefits 
(Alston,  Craig,  and  Pardey,  1998). 

•  the  difficulty  in  attributing  outcome  effects  to  particular  research  causes.  The 
connection  between  research  and  effects  is  diffuse  and  indirect.  Impacts  require 
inputs  other  than  just  the  conclusion  of  a  research  and  development  (R&D)  proj¬ 
ect  and  dissemination  (Rip,  2001).  The  quality  of  research,  the  extent  to  which 
the  research  is  diffused  to  those  in  a  position  to  use  this  information  to  generate 
impacts,  and  the  ability  of  research  users  to  extract  full  value  from  it  will  all  influ¬ 
ence  the  final  impact  of  research  (Allen  Consulting  Group  and  Australia  Depart¬ 
ment  of  Education,  Science,  and  Training,  2005). 

•  the  difficulty  of  collecting  evidence  of  impact  without  undue  selection  bias  for 
positive  impact 

•  lack  of  well-articulated  program  theory  that  describes  the  path  of  how  a  program 
achieves  its  end  outcomes. 
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Despite  these  factors,  approaches  to  evaluating  the  impact  of  research  have  evolved 
substantially  in  the  past  decade.1  Techno  metrics,  sociometrics,  biblio  metrics,  value¬ 
mapping,  expert  review,  and  case  studies  represent  both  quantitative  and  qualitative 
means  of  assessing  the  benefits  of  research  to  industry,  government,  or,  more  broadly, 
the  public  (Donovan,  2008).  In  1994,  a  special  issue  of  Evaluation  Review  discussed 
the  strengths  and  weaknesses  associated  with  current  methods  of  assessing  research 
impact.  The  authors  concluded  that,  because  research  impact  has  many  facets,  its  assess¬ 
ment  must  use  as  many  methods,  types  of  expertise,  and  approaches  as  are  required  to 
address  these  different  facets  (Kostoff,  Averch,  and  Chubin,  1994).  Different  approaches 
that  result  in  similar  findings  engender  confidence  in  the  overall  results.  Different 
approaches  that  produce  conflicting  results  still  have  value  and  increase  understanding 
when  trying  to  determine  the  causes  of  the  differences  and  resolve  them.  An  important 
objective  of  this  special  issue  was  to  increase  interest  in  implementing  research  impact 
assessments.  As  recently  as  summer  2008,  the  journal  New  Directions  for  Evaluation 
covered  the  topic  of  assessing  scientific  research  and  acknowledged  the  need  for  new 
and  improved  methods  of  assessment,  particularly  those  that  can  increase  the  quality 
of  research  that  is  done,  reduce  the  cost  of  doing  it,  and  lend  public  credibility  to  the 
manner  in  which  it  is  funded  (Coryn  and  Scriven,  2008). 

Interest  in  research  impact  evaluations  has  also  been  spurred  by  the  governments’ 
desire  to  gauge  the  value  of  publicly  funded  research  for  budget  and  resource  alloca¬ 
tion  decisions,  both  nationally  and  internationally.  In  the  United  States,  the  Govern¬ 
ment  Performance  and  Results  Act  (GPRA)  and  the  Program  Assessment  Rating  Tool 
(PART)  are  the  most  recent  manifestations  of  the  public’s  concern  about  the  payoff  of 
federally  funded  research  and  have  spurred  growing  interest  in  focusing  on  the  impact 
of  research  programs  and,  more  specifically,  on  outcome  measures  as  a  way  of  monitor¬ 
ing  and  determining  impact. 

Passed  by  Congress  in  1993,  GPRA’s  purpose  is  to  improve  the  efficiency  and 
effectiveness  of  federal  programs  by  establishing  a  system  to  set  goals  for  program 
performance  and  to  measure  results  (Cozzens,  1996).  Under  this  law,  federal  agencies 
must  submit,  to  the  Office  of  Management  and  Budget  (OMB)  and  the  Congress, 
long-range  strategic  goals,  beginning  with  the  fiscal  year  (FY)  1999  budget.  GPRA 
requires  that  agencies  identify  goals  for  specific  outcomes  of  their  activities,  develop 
performance  measures  to  assess  programmatic  outputs  and  outcomes,  gather  the  req¬ 
uisite  data  to  evaluate  performance  measures,  and  report  annually  on  progress  toward 
goals.  PART,  which  the  George  W.  Bush  administration  introduced  in  2002  to  further 
the  policy  objectives  of  budget  and  performance  integration,  is  a  systematic  method  of 
assessing  program  performance  across  the  federal  government  (OMB,  2008).  It  includes 
a  series  of  diagnostic  questions  designed  to  provide  a  consistent  approach  to  the  evalu- 


1  The  literature  on  evaluating  the  impact  of  research  is  extensive.  An  overview  of  this  field  can  be  found  in  Boze¬ 
man  and  Melkers  (1993). 
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ation  of  federal  programs.2  Both  PART  and  GPRA  represent  federal  efforts  to  achieve 
more  accountability  for  program  outcomes.  However,  PART  is  much  more  explicit  in 
calling  for  information  on  achieving  outcomes,  including  independent  evaluation  as  a 
method  of  determining  impact.  Evaluations  are  covered  in  two  of  the  four  sections  of 
PART.  Question  2.6,  which  is  part  of  the  Strategic  Planning  section,  asks,  “Are  inde¬ 
pendent  evaluations  of  sufficient  scope  and  quality  conducted  on  a  regular  basis  or  as 
needed  to  support  program  improvements  and  evaluate  effectiveness  and  relevance  to 
the  problem,  interest,  or  need?”  In  section  IV,  Program  Results/Accountability,  ques¬ 
tion  4.5  asks,  “Do  independent  evaluations  of  sufficient  scope  and  quality  indicate  that 
the  program  is  effective  and  achieving  results?”  Evidence  to  support  an  answer  of  “yes” 
for  question  2.6  should  adhere  to  the  following  guidance: 

To  receive  a  Yes,  agencies  must  demonstrate  that  they  have  chosen  and  applied 
evaluation  methods  that  provide  the  most  rigorous  evidence  of  a  program’s  effec¬ 
tiveness  that  is  appropriate  and  feasible.  A  program  may  satisfy  this  criterion  if 
the  agency  and  OMB  determine  that  the  program  is  in  the  process  of  developing 
new  evaluation  approaches  that  will  provide  the  most  rigorous  evidence  possible 
by  a  specified  future  date.  The  most  significant  aspect  of  program  effectiveness  is 
impact — the  outcome  of  the  program,  which  otherwise  would  not  have  occurred 
without  the  program  intervention.  (OMB,  2008,  p.  30) 

Questions  2.6  and  4.5  are  clearly  linked.  Thus,  in  addition  to  meeting  the  crite¬ 
ria  outlined  for  question  2.6,  evidence  to  support  an  answer  of  “yes”  for  question  4.5 
should  take  note  of  the  following: 

Relevant  evaluations  would  be  at  the  national  program  level,  rather  than  evalua¬ 
tions  of  one  or  more  program  partners,  and  would  not  focus  only  on  process  indi¬ 
cators  such  as  the  number  of  grants  provided,  or  hits  on  a  web  site.  .  .  .  Evidence 
should  include  a  summary  discussion  of  the  findings  of  an  evaluation  conducted 
by  academic  and  research  institutions,  agency  contracts,  other  independent  enti¬ 
ties,  the  Government  Accountability  Office,  or  Inspectors  General.  (OMB,  2008, 
pp.  59-60) 

Although  PART  guidance  points  to  the  randomized  controlled  trial  (RCT)  as  an 
example  of  the  best  type  of  evaluation  to  demonstrate  actual  program  impact,  RCTs 
are  not  suitable  for  every  program  and  can  generally  be  employed  only  under  very  spe¬ 
cific  circumstances  (OMB,  2004).  When  assessment  of  program  effectiveness  is  subject 
to  high  uncertainty,  expert  evaluation,  a  form  of  peer  review,  should  be  considered. 


2  There  is  no  standard  definition  for  the  term  program.  For  purposes  of  PART,  the  OMB  describes  the  unit  of 
analysis  ( program )  as  (1)  an  activity  or  set  of  activities  clearly  recognized  as  a  program  by  the  public,  OMB,  or 
Congress;  (2)  having  a  discrete  level  of  funding  clearly  associated  with  it;  and  (3)  corresponding  to  the  level  at 
which  budget  decisions  are  made. 
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Program  uncertainty  can  be  defined  and  identified  by  the  following  situations  and  con¬ 
ditions  (Averch,  2004,  p.  293): 

•  A  public  agency  has  been  operating  a  “program”  for  a  number  of  years,  and  it 
cannot  be  certain  about  the  effective  quantity  or  quality  of  inputs  it  has  bought 
during  those  years,  and  there  is  no  clear  way  to  measure  these. 

•  The  expected  “benefits”  or  “outcomes”  of  the  program  are  highly  uncertain  in  the 
present  or  must  occur  in  the  future. 

•  The  agency  does  not  know  with  precision  whether  decision-relevant  outcomes  can 
be  attributed  to  the  inputs  and  the  design  of  the  program. 

For  many  federal  research  programs,  at  least  one  (if  not  all)  of  these  conditions 
is  operative.  Thus,  most  evaluations  of  research  impact  rely  on  some  form  of  expert 
review.  In  the  following  section,  we  provide  a  brief  discussion  of  expert  review  and 
describe  how  it  has  been  used  to  evaluate  research  impact. 


Expert  Review 

In  1999,  the  National  Academies  offered  the  following  recommendations  for  the  evalu¬ 
ation  of  basic  and  applied  research: 

The  most  effective  way  to  evaluate  research  programs  is  by  expert  review.  The 
most  commonly  used  form  of  expert  review  of  quality  is  peer  review.  This  operates 
on  the  premise  that  the  people  best  qualified  to  judge  the  quality  of  research  are 
experts  in  the  field  of  research.  This  premise  prevails  across  the  research  spectrum 
from  basic  research  to  applied  research.  (Committee  on  Science,  Engineering,  and 
Public  Policy,  1999,  pp.  9-10) 

Most  federal  agencies  typically  use  several  types  of  expert-review  methods,  includ¬ 
ing  (1)  peer  review,  which  is  commonly  used  to  make  judgments  about  the  careers  of 
individual  staff  members,  the  value  of  publications,  the  standing  of  institutions,  and 
the  allocation  of  funds  to  individuals,  organizations,  and  fields  of  inquiry;  (2)  relevance 
review,  which  is  used  to  judge  whether  an  agency’s  programs  are  relevant  to  its  mis¬ 
sion;  and  (3)  benchmarking,  which  is  used  to  evaluate  the  standing  of  an  organization, 
program,  or  facility  relative  to  another  (Ruegg  and  Feller,  2003).  However,  in  the  past 
few  decades,  wide-scale  use  of  peer-review  panels  has  become  commonplace  to  evalu¬ 
ate  larger  units,  such  as  research  groups,  institutes,  and  research  programs.  In  addition 
to  scientific  merit,  these  large-scale  panels  are  often  concerned  with  the  socioeconomic 
impact  of  research  (Coryn  and  Scriven,  2008).  Thus,  increasingly,  the  term  peer  review 
has  been  reserved  for  the  more  traditional  review  and  assessment  systems  of  scholarly 
communities,  such  as  reviews  of  manuscripts  for  journals,  and  the  term  expert  panel 
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evaluation  or  expert  judgment  is  used  for  the  evaluations  of  research  that  go  beyond  the 
merit  of  individual  instances  or  pieces  of  research,  or  of  individual  researchers  (Coryn 
and  Scriven,  2008). 

There  are  two  types  of  expert-panel  evaluation:  peer-panel  evaluation  and  mixed- 
panel  evaluation  (Langfeldt,  2002).  One  or  the  other  is  commissioned  often  ad  hoc,  for 
evaluation  at  the  program,  institutional,  or  discipline  level.  Peer-panel  evaluation  uses 
only  researchers  qualified  in  the  subject  matter  under  review.  When  the  expert-panel 
evaluation  consists  of  both  peers  and  other  experts  (for  instance,  experts  on  policy  or 
commercialization  of  research),  it  is  referred  to  as  mixed-panel  evaluation. 

Panel  reviews  are  the  main  method  for  evaluating  the  impact  of  European  Union 
(EU)  research  programs  (Arnold,  Muscio,  and  Zaman,  2005).  There  have  been  a 
number  of  recent  reviews  exploring  alternative  methods  for  evaluating  EU  investments, 
such  as  bibliometrics,  econometrics,  and  social  analysis.  However,  the  value  of  panels 
continues  to  be  emphasized,  and  other  methods  have  been  advocated  as  a  complement 
rather  than  a  replacement  (Boaz,  Fitzpatrick,  and  Shaw,  2008).  In  the  United  States, 
mission  agencies,  such  as  the  departments  of  Defense,  Energy,  and  Agriculture,  have 
used  expert-panel  review  to  strengthen  their  program-review  processes  since  the  1980s 
and  1990s  (Cozzens,  1999).  These  expert  panels  rely  on  the  review  of  relevant  materials 
on  program  activities  and  results  to  help  them  assess  the  extent  to  which  research  pro¬ 
grams  have  had  impact.  Thus,  research  programs  bear  much  of  the  responsibility  for 
gathering  and  presenting  data  that  provide  evidence  for  the  impact  of  their  programs. 
Demonstrating  and  communicating  impacts  for  the  purposes  of  expert-panel  evalua¬ 
tion  is  subject  to  many  of  the  same  methodological  hurdles  as  evaluating  research.  The 
following  section  describes  some  of  the  issues  associated  with  them. 


Demonstrating  and  Communicating  the  Impact  of  Research  Programs 

The  impact  of  research  programs  is  assessed  largely  outside  of  the  context  of  the 
researcher’s  scientific  discipline  and  relies  on  the  response  of  customers,  stakeholders, 
and  others  to  outputs  from  the  research  program.  As  such,  it  requires  research  pro¬ 
grams  to  do  more  than  document  research  outputs,  such  as  research  publications,  jour¬ 
nal  articles,  or  conference  presentations.  Demonstrating  impact  involves  following  the 
trail  of  research  beyond  the  production  of  outputs  and  gathering  information  that  pro¬ 
vides  an  “evidence  base”  to  support  claims  of  impact.  While  most  research  programs 
may  be  familiar  and  most  comfortable  with  presenting  outputs,  it  is  more  difficult  to 
determine  and  collect  the  data  needed  on  outcomes.  For  example,  if  an  output  from  a 
research  program  is  a  stakeholder  presentation,  part  of  demonstrating  impact  is  articu¬ 
lating  the  route  of  use  of  the  presentation  that  contributed  to  changes  in  workplace 
practices.  Tracing  this  route  requires  an  understanding  of  the  program  theory  of  how 
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the  research  is  intended  to  achieve  outcomes,  which  involves  research  transfer  and  use, 
rather  than  just  production. 

Communicating  research  impacts  to  an  audience  that  includes  nonresearchers 
can  also  present  challenges.  Many  research  programs  are  not  accustomed  to  reporting 
to  audiences  outside  of  the  scientific  community.  Communicating  impact  to  audiences 
who  are  interested  primarily  in  how  research  activities  have  served  a  broader  com¬ 
munity  is  different  from  writing  for  a  research  audience.  Adapting  the  content,  struc¬ 
ture,  and  writing  style  for  a  more  diverse  audience  is  critical  for  making  the  case  that 
research  activities  have  contributed  to  outcomes. 

As  researchers  are  increasingly  required  to  demonstrate  the  impact  of  their  work, 
the  number  of  specialized  tools  and  models  that  offer  useful  ways  of  conceptualizing 
and  describing  impact  has  increased.  Though  many  different  frameworks  are  discussed 
in  the  literature,  only  a  few  are  actually  used  in  impact  evaluations  (Boaz,  Fitzpatrick, 
and  Shaw,  2008).  The  Buxton-Hanney  payback  framework  of  health  research  benefits, 
which  consists  of  a  five-category  classification  system  for  the  benefits  of  research,  is  one 
such  framework  (Buxton  and  Hanney,  1996).  The  research  impact  framework,  devel¬ 
oped  by  researchers  at  the  London  School  of  Hygiene  and  Tropical  Medicine,  is  another 
one  (Kuruvilla  et  ah,  2006).  This  framework  covers  a  wide  range  of  potential  areas  of 
health-research  impact  and  standardizes  ways  of  describing  them,  so  that  individual 
researchers  without  any  specific  training  in  research  impact  assessment  could  use  the 
framework  to  describe  the  impact  of  their  work.  Finally,  Australia’s  research-quality 
framework  (RQF)  is  a  research  evaluation  model  that  considers  research  impacts  in 
addition  to  the  conventional  quality  measures  normally  used  in  the  academic  commu¬ 
nity.  The  RQF  was  proposed  as  a  panel-based  exercise  to  evaluate  both  research  excel¬ 
lence  and  the  wider  benefits  of  academic  research  for  the  nation  and  to  allocate  funds 
on  the  basis  of  outcomes  (Donovan,  2008). 


Purpose  of  This  Book 

In  September  2004,  the  National  Institute  for  Occupational  Safety  and  Health 
(NIOSH)  contracted  with  the  National  Academies  to  conduct  reviews  of  up  to  15  of 
its  research  programs.  Between  2005  and  2008,  eight  such  reviews  were  conducted. 
The  purpose  of  the  reviews  was  to  judge  the  extent  to  which  NIOSH  science  is  (1)  rel¬ 
evant  to  real-world  occupational  safety  and  health  (OSH)  problems;  (2)  meets  the 
highest  scientific  quality  standards  for  which  it  can  strive;  and  (3)  achieves  the  great¬ 
est  impact  that  it  possibly  can  (see  Howard,  2005).  The  National  Academies’  external 
review  relies  on  the  use  of  expert  panels  to  evaluate  the  impact  and  relevance  of  each  of 
the  research  programs.  To  assist  in  preparation  for  the  reviews,  NIOSH  asked  RAND 
to  develop  and  apply  a  methodology  for  preparing  for  external  program  reviews.  As 
a  result  of  our  work  with  NIOSH,  we  have  developed  a  set  of  tools  that  research  pro- 
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grams  generally  can  apply  to  demonstrate  and  communicate  impact.  Collectively,  these 
tools  offer  research  programs  a  methodology  for  conceptualizing  their  research  path¬ 
ways  to  outcomes,  tracing  specific  cases  of  outcomes  back  to  research  activities,  and 
creating  a  model  package  that  concisely  communicates  the  evidence  of  impact.  This 
book  describes  the  development  and  use  of  each  of  these  tools.  Because  it  was  written 
prior  to  the  completion  of  all  of  the  reviews,  this  book  does  not  include  feedback  from 
the  programs  on  the  tools  or  an  assessment  of  the  overall  effectiveness  of  these  tools 
in  helping  to  prepare  for  external  reviews.  Moreover,  it  does  not  offer  commentary  on 
NIOSH’s  overall  experience  in  the  external  review  process  or  a  critique  of  the  National 
Academies’  review  process.  Both  of  these  topics  are  beyond  the  scope  of  this  book. 


Organization  of  This  Book 

The  remainder  of  this  book  is  organized  as  follows.  Chapter  Two  describes  the  struc¬ 
ture  of  the  National  Academies’  external  review  and  the  criteria  used  to  assess  research 
impact  and  relevance.  Chapter  Three  focuses  on  our  core  tool,  the  logic  model,  which 
we  developed  with  each  of  the  NIOSH  research  programs.  The  logic  model  depicted 
the  path  from  research  activities  to  outcomes  and  was  designed  to  facilitate  the  review¬ 
ers’  understanding  of  how  the  programs  intended  to  achieve  their  end  outcomes  and 
demonstrate  impact.  Chapter  Four  describes  the  outcome  worksheet,  which  was  used 
to  help  the  research  programs  trace  specific  outcomes  back  to  research  activities.  This 
backward  tracing  is  based  on  historical  tracing,  and,  in  Chapter  Four,  we  discuss  how 
this  methodology  has  evolved  and  how  we  used  it  to  map  research  impact.  Chapter 
Five  focuses  on  communicating  impact  and  describes  the  structure  and  contents  of  the 
evidence  packages  that  each  research  program  submitted  to  the  expert  panel.  A  key 
component  of  the  evidence  package  was  the  outcome  narrative,  which  was  used  to  tell 
the  research  story  of  impact.  Finally,  in  Chapter  Six,  we  draw  conclusions  and  discuss 
the  utility  of  these  tools  beyond  preparation  for  external  review. 


CHAPTER  TWO 


Using  Expert  Evaluation  to  Measure  the  Impact  of  Federal 
Programs:  The  National  Academies'  Review  of  NIOSH 


In  this  chapter,  we  provide  a  brief  overview  of  NIOSH  and  describe  the  structure  of 
the  National  Academies’  external  review  and  the  criteria  that  were  used  to  evaluate 
the  agency’s  research  programs.  In  addition,  we  also  describe  RAND’s  role  in  assisting 
NIOSH. 


Overview  of  the  National  Institute  of  Occupational  Safety  and  Health 

NIOSH  is  the  federal  agency  responsible  for  conducting  research  and  making  rec¬ 
ommendations  for  the  prevention  of  work-related  injury  and  illness  (see  NIOSH, 
undatedja]).  NIOSH  is  part  of  the  Centers  for  Disease  Control  and  Prevention  (CDC) 
in  the  U.S.  Department  of  Health  and  Human  Services  (DHHS).  The  main  legislative 
underpinnings  of  NIOSH  are  the  Federal  Coal  Mine  Health  and  Safety  Act  of  1969 
(Pub.  L.  No.  91-173,  amended  by  Pub.  L.  No.  95-164  in  1977  or  MSHAct;  also  known 
as  the  Coal  Act)  and  the  Occupational  Safety  and  Health  Act  of  1970  (Pub.  L.  91-596; 
also  known  as  the  OSHAct).  The  Coal  Act  was  passed  in  the  aftermath  of  a  devastating 
coal-mine  explosion  that  occurred  in  Farmington,  West  Virginia,  in  1968.  It  took  the 
lives  of  78  miners  and  crystallized  public  opinion  that  stronger  measures  were  needed 
to  protect  coal  miners  at  work.  Activities  required  by  the  Coal  Act  were  split  between 
the  U.S.  Department  of  Health,  Education,  and  Welfare  (HEW;  now  DHHS),  which 
engaged  in  nonregulatory  activities,  such  as  health  screening  and  research,  and  the 
Mine  Enforcement  and  Safety  Administration  (MESA)  in  the  U.S.  Department  of 
the  Interior  (DOI),  which  engaged  in  developing  and  enforcing  workplace  safety  and 
health  regulations  in  the  mining  industry.  After  its  creation  by  the  OSHAct,  NIOSH 
assumed  the  health  screening  and  research  responsibilities  specified  under  the  Coal 
Act.  When  the  Coal  Act  was  amended  in  1977,  the  Mine  Safety  and  Health  Adminis¬ 
tration  (MSHA)  in  the  U.S.  Department  of  Labor  (DOL)  replaced  MESA. 

The  OSHAct  created  NIOSH  and  the  Occupational  Safety  and  Health  Adminis¬ 
tration  (OSHA).  OSHA  is  in  the  DOL  and  is  responsible  for  developing  and  enforcing 
workplace  safety  and  health  regulations.  NIOSH,  in  DHHS,  was  established  to  help 
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ensure  safe  and  healthful  working  conditions  by  providing  research,  information,  edu¬ 
cation,  and  training  in  the  OSH  held. 

The  main  organizational  units  of  NIOSH  are  divisions,  laboratories,  and  offices 
(see  Figure  2.1).  These  are  a  mixture  of  disease-  and  injury-specific  divisions  (respira¬ 
tory  diseases,  safety  research),  expertise-specific  divisions  (applied  research  and  tech¬ 
nology,  laboratory  research,  surveillance  and  held  studies,  education  and  information 
dissemination),  and  industry-specihc  units  (mining). 

The  divisions,  laboratories,  and  offices  are  geographically  dispersed  in  Cincinnati, 
Ohio;  Morgantown,  West  Virginia;  Pittsburgh,  Pennsylvania;  Spokane,  Washington; 
Denver,  Colorado;  and  Anchorage,  Alaska.  NIOSH  leadership  is  located  in  Washing¬ 
ton,  D.C.,  and  Atlanta. 

As  shown  in  Table  2.1,  the  NIOSH  research  program  portfolio  is  organized  into 
eight  sector  programs  that  represent  industrial  sectors  and  24  cross-sector  programs 
organized  around  adverse  health  outcomes,  statutory  programs,  and  global  efforts 
(NIOSH,  2008a). 

Research  programs  are  spread  across  the  organizational  units  in  NIOSH.  Thus, 
a  research  program  is  based  on  a  matrix  approach,  rather  than  a  unit-based  approach, 
and  is  not  an  identihable  entity  in  the  NIOSH  organizational  chart.  For  example,  the 
research  activities  in  the  Hearing  Loss  Prevention  program  are  conducted  in  at  least 
four  different  divisions  of  NIOSH.  Preparation  for  external  reviews  of  these  matrixed 

Figure  2.1 

NIOSH  Organizational  Chart 


SOURCE:  DHHS  (2008). 
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Table  2.1 

NIOSH  Research  Program  Portfolio 


Type 

Program 

NORA  sector 

Agriculture,  Forestry,  and  Fishing 

Construction 

Healthcare  and  Social  Assistance 

Manufacturing 

Mining  (includes  Oil  and  Gas  Extraction  subsector) 

Services 

Transportation,  Warehousing,  and  Utilities 

Wholesale  and  Retail  Trade 

NIOSH  cross-sector 

Authoritative  Recommendations 

Cancer,  Reproductive,  and  Cardiovascular  Diseases 

Communications  and  Information  Dissemination 

Economics 

Emergency  Preparedness  and  Response 

Engineering  Controls 

Exposure  Assessment 

Global  Collaborations 

Health  Hazard  Evaluation  (HHE) 

Hearing  Loss  Prevention 

Immune  and  Dermal  Diseases 

Musculoskeletal  Disorders 

Nanotechnology 

Occupational  Health  Disparities 

Personal  Protective  Technology  (PPT) 

Prevention  Through  Design 

Radiation  Dose  Reconstruction 

Respiratory  Diseases 

Small  Business  Assistance  and  Outreach 

Surveillance 

Training  Grants 
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Table  2.1 — Continued 


Type 

Program 

NIOSH  cross-sector 
(continued) 

Traumatic  Injury 

Work  Organization  and  Stress-Related  Disorders 

Worklife  Initiative 

SOURCE:  NIOSH  (2008a 

,  p.  16). 

programs  required  the  programs  to  gather  information  from  across  the  organizational 
units.  Another  issue  was  program  overlap.  Some  research  activities  (e.g.,  research  to 
develop  standardized  methods)  were  logically  reportable  (and  sometimes  actually 
reported)  by  more  than  one  program. 


Expert  Review  by  the  National  Academies 

The  NIOSH  charge  to  the  National  Academies  was  to  review  its  programs  to  assess 
their  contributions  to  improving  health  and  safety  in  the  workplace.  Specifically,  the 
National  Academies  were  asked  to  evaluate  impact,  relevance,  and  progress  in  identify¬ 
ing  emerging  research  areas  through  the  following  tasks: 

•  assessment  of  the  program’s  contribution,  through  OSH  research,  to  reductions 
in  workplace  hazardous  exposures,  illnesses,  or  injuries  through 

-  an  assessment  of  the  relevance  of  the  program’s  activities  to  the  improvement 
of  OSH 

-  an  evaluation  of  the  impact  that  the  program’s  research  has  had  in  reducing 
work-related  hazardous  exposures,  illnesses,  and  injuries 

•  assessment  of  the  program’s  effectiveness  in  targeting  new  research  areas  and 
identifying  emerging  issues  in  OSH  most  relevant  to  future  improvements  in 
workplace  protection. 

In  response  to  the  charge,  the  Institute  of  Medicine  (IOM)  and  the  Division  of 
Earth  and  Life  Studies  (DELS)  of  the  National  Academies  conducted  evaluations  of 
eight  programs.  Each  program  was  reviewed  by  a  separate  evaluation  committee  (EC) 
composed  of  persons  with  expertise  appropriate  to  evaluating  the  specific  NIOSH 
research  programs;  in  some  cases,  this  included  representatives  of  stakeholder  groups 
(such  as  labor  unions  and  industry)  and  experts  in  technology  transfer  and  program 
evaluation.  In  conducting  their  evaluations,  the  ECs  ascertained  whether  NIOSH  is 
doing  the  right  things  (relevance)  and  whether  those  things  are  improving  health  and 
safety  in  the  workplace  (impact). 
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The  ECs  could  choose  among  three  general  rime  frames  for  rheir  review:  (1)  1970- 
1995,  (2)  1996-2005,  or  (3)  after  2005.  These  time  frames  are  associated  with  the 
National  Occupational  Research  Agenda  (NORA).  NORA  is  a  partnership  program 
to  stimulate  innovative  research  and  improved  workplace  practices.  Unveiled  in  1996, 
NORA  is  an  effort  to  guide  and  coordinate  research  nationally,  not  only  for  NIOSH 
but  for  the  entire  OSH  community  (NIOSH,  2008b).  Diverse  parties  collaborated  to 
identify  the  most-critical  issues  in  workplace  safety  and  health.  Partners  then  worked 
together  to  develop  goals  and  objectives  for  addressing  those  needs.  Thus,  the  earliest 
period  (1970-1995)  is  considered  the  pre-NORA  time  frame  and  represents  the  period 
from  the  founding  of  NIOSH  to  the  initiation  of  the  NORA  process.  The  second  and 
third  time  frames  refer  to  the  first  and  second  decades  of  NORA,  respectively. 

As  part  of  their  review,  the  ECs  conducted  information-gathering  sessions  to 
obtain  information  from  the  NIOSH  research  program  that  was  being  reviewed, 
stakeholders  directly  affected  by  the  NIOSH  research,  and  relevant  independent  par¬ 
ties.  For  example,  in  the  review  of  the  Mining  program,  the  EC  interacted  with 
more  than  40  Mining  program  employees  and  heard  17  presentations  by  NIOSH 
and  the  Mining  program  and  nine  stakeholder  presentations  during  an  open-session 
meeting.  This  included  a  presentation  from  the  acting  director  of  MSHA,  labor- 
union  representatives,  equipment  manufacturers,  and  training  consultants.  On  aver¬ 
age,  each  EC  consisted  of  10  members  and  met  three  times  during  the  course  of  the 
review.  To  provide  the  final  assessment  of  the  research  program’s  impact  and  rel¬ 
evance  (charge  1),  the  ECs  rated  the  program’s  performance  on  a  five-point  scale  in 
which  1  is  the  lowest  and  5  is  the  highest  rating.1  Only  single-integer  values  could  be 
assigned.  In  addition  to  the  numerical  scores,  final  program  ratings  were  supported 
by  reviewers’  explanations  of  the  scores.  Although  the  research  programs  were  not 
rated  on  charge  2,  the  ECs  responded  to  this  charge  by  providing  a  qualitative,  nar¬ 
rative  assessment  of  the  program’s  efforts  and  suggestions  about  emerging  issues  that 
the  program  should  be  prepared  to  address.  At  the  conclusion  of  the  review,  the  ECs 
prepared  a  final  report  for  NIOSH  that  followed  the  standard  template  shown  in 
Table  2.2. 


The  National  Academies'  Evaluation  Framework 

To  maintain  consistency  in  the  evaluation  across  the  independent  ECs,  the  National 
Academies  appointed  a  committee  of  14  members  (the  Framework  Committee),  includ¬ 
ing  persons  with  expertise  in  occupational  medicine  and  health,  industrial  health  and 
safety,  industrial  hygiene,  epidemiology,  civil  and  mining  engineering,  sociology, 


i 


Tables  2.3  and  2.4  provide  full  descriptions  of  each  numerical  value. 
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Table  2.2 

Suggested  Outline  for  Evaluation  Committee  Reports 


Section 

Description 

1.  Introduction 

This  section  should  be  a  brief  descriptive  summary  of  the  history  of  the  program 
(and  subprograms)  being  evaluated  with  respect  to  pre-NORA,  NORA  1,  and 
current  and  future  plans  of  the  research  program  presented  by  NIOSH.  It  should 
present  the  context  for  the  research  on  safety  and  health;  goals,  objectives, 
and  resources;  groupings  of  subprograms;  and  any  other  important  pertinent 
information. 

II.  Evaluation  of 
Programs  and 
Subprograms 
(charge  1) 

A.  Evaluation  summary  (should  include  a  brief  summary  of  the  evaluation  with 
respect  to  impact  and  relevance,  scores  for  impact  and  relevance,  and  summary 
statements) 

B.  Strategic  goals  and  objectives  (should  describe  assessment  of  the  program 
and  subprograms  for  relevance) 

C.  Review  of  inputs  (should  describe  adequacy  of  inputs  to  achieve  goals) 

D.  Review  of  activities  (should  describe  assessment  of  the  relevance  of  the 
activities) 

E.  Review  of  research  program  outputs  (should  describe  assessment  of  relevance 
and  potential  usefulness  of  the  research  program) 

F.  Review  of  intermediate  outcomes  and  causal  impact  (should  describe 
assessment  of  the  intermediate  outcomes  and  the  attribution  to  NIOSH  and 
include  the  likely  impacts  and  recent  outcomes  in  the  assessment) 

G.  Review  of  end  outcomes  (should  describe  the  end  outcomes  related  to  health 
and  safety  and  provides  an  assessment  of  the  type  and  degree  of  attribution  to 
NIOSH) 

H.  Review  of  other  outcomes  (should  discuss  health  and  safety  impacts 
that  have  not  yet  occurred;  beneficial  social,  economic,  and  environmental 
outcomes;  and  international  dimensions  and  outcomes) 

I.  Summary  of  ratings  and  rationale 

III.  NIOSH  Targeting 
of  New  Research 
and  Identification 
of  Emerging  Issues 
(charge  2) 

The  EC  should  assess  the  progress  that  the  NIOSH  program  has  made  in 
targeting  new  research  in  OSH  and  whether  the  NIOSH  program  has  identified 
emerging  issues  that  appear  especially  important  in  terms  of  relevance  to  the 
mission  of  NIOSH  and  should  respond  to  NIOSH's  perspective  and  add  its  own 
recommendations. 

IV.  Recommendations 
for  Program 
Improvement 

On  the  basis  of  the  review  and  evaluation  of  the  program,  the  EC  may  provide 
recommendations  for  improving  the  relevance  of  the  NIOSH  research  program 
to  health  and  safety  conditions  in  the  workplace  and  the  impact  of  the  research 
program  on  health  and  safety  in  the  workplace. 

SOURCE:  National  Academies  (2007,  p.  35). 

program  evaluation,  communication,  and  toxicology;  representatives  of  industry  and 
of  the  workforce;  and  a  scientist  experienced  in  international  occupational-health  issues 
to  develop  a  guide  and  provide  a  common  structure  for  the  review  of  the  different 
research  programs.  The  ensuing  document,  called  the  framework  document  (National 
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Academies,  2007), 2  defines  the  evaluation  framework  developed  by  the  Framework 
Committee  and  outlines  the  evaluation  criteria  to  be  used  by  the  ECs,  the  information 
needs,  and  the  specific  evaluation  questions  to  be  considered. 

The  Framework  Committee  used  the  NIOSH  logic  model  (see  Appendix  A)  to 
develop  a  flow  chart  to  outline  the  scope  and  steps  of  an  EC  evaluation.3  The  flow  chart 
breaks  the  NIOSH  logic  model  into  discrete,  sequential  program  components  to  be 
characterized  or  assessed  by  the  ECs.  These  components  are  summarized  here:4 

•  Review  of  major  program-area  challenges:  independent  assessment  by  EC  members 
to  compare  with  NIOSH  program-area  goals 

•  Review  of  strategic  goals  and  objectives  driving  current  program:  assessment  of 
NIOSH  process  to  select  program  goals,  evaluation  of  goals  selected  by  NIOSH, 
comparison  with  EC  assessment  of  challenges 

•  Review  of  inputs:  assessment  of  inputs  consisting  of  planning  and  production  inputs 
and  including  EC  consideration  of  the  degree  to  which  allocation  of  funding  and 
personnel  was  commensurate  with  resources  needed  to  conduct  the  research 

•  Review  of  activities:  assessment  of  activities  including  both  research  and  transfer 
activities — the  former  to  determine  whether  they  are  consistent  with  program 
goals,  objectives,  and  major  challenges  of  the  research  program,  and  the  latter  to 
determine  whether  the  program  appropriately  targets  outputs  in  a  manner  that 
will  have  the  greatest  impact 

•  Review  of  outputs:  qualitative  assessment  of  relevance  and  utility  of  outputs;  the 
outputs  of  a  highly  ranked  program  will  address  needs  in  high-priority  areas,  con¬ 
tain  new  knowledge  or  technology  that  is  effectively  communicated,  contribute 
to  capacity-building  inside  and  outside  NIOSH,  and  be  relevant  to  the  pertinent 
populations 

•  Review  of  intermediate  outcomes:  review  of  responses  by  NIOSH  stakeholders 
to  NIOSH  products,  such  as  how  widely  products  have  been  used  or  programs 
implemented  or  whether  products  have  resulted  in  changes  in  the  workplace  or 
facilities  by  its  peers 

•  Review  of  end  outcomes:  assessment  of  NIOSH’s  contribution  to  improvements 
in  workplace  health  and  safety,  including  decreases  in  injuries,  illnesses,  deaths, 
and  exposures  to  risk;  if  there  is  no  direct  evidence  of  improvements  in  health 


2  This  is  the  second  and  most  recent  version  of  the  framework  document  that  was  used  for  NIOSH  program 
reviews.  The  first  version,  dated  December  19,  2005,  was  developed  during  the  reviews  of  the  Hearing  Loss  Pre¬ 
vention  and  Mining  programs. 

3  The  NIOSH  logic  model  can  be  found  in  Appendix  A.  Logic  models  are  defined  and  discussed  in  detail  in 
Chapter  Three. 

4  The  complete  discussion  of  each  program  component  can  be  found  in  the  National  Academies  (2007)  frame¬ 
work  document. 
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and  safety,  intermediate  outcomes  may  be  used  as  proxies  for  end  outcomes  as 
long  as  the  ECs  can  qualify  their  findings 
•  Review  of  potential  outcomes:  assessment  of  other  outcomes,  including  beneficial 
changes  that  have  not  yet  occurred;  social,  economic,  security,  or  environmental 
outcomes;  and  the  impact  that  NIOSH  has  had  on  international  occupational 
safety  and  health. 


Criteria  for  Assessing  Impact  and  Relevance 

The  ECs  were  expected  to  use  their  expert  judgment  to  rate  the  relevance  and  impact 
of  the  overall  research  program.  In  assessing  overall  impact,  the  framework  doc¬ 
ument  recommended  that  the  ECs  consider  the  available  evidence  of  changes  in 
work-related  risks  and  adverse  effects  and  external  factors  related  to  the  changes. 
Moreover,  the  ECs  were  advised  to  evaluate  the  impact  of  the  research  activities 
separately  from  the  impact  of  the  transfer  activities.  Transfer  activities  occur  in  two 
contexts:  NIOSEI  efforts  to  translate  intellectual  products  into  practice  and  stake¬ 
holder  efforts  to  integrate  NIOSH  results  into  the  workplace  (National  Academies, 
2007).  High-impact  assessments  required  the  EC’s  judgment  that  the  research  pro¬ 
gram  had  contributed  to  outcomes — for  example,  the  determination  that  outcomes 
would  have  occurred  earlier  or  are  better  than  they  would  have  in  the  absence  of 
the  research  program  or  that  outcomes  would  have  occurred  were  it  not  for  external 
factors  beyond  NIOSH’s  control  or  ability  to  plan  around.  The  criteria  for  scoring 
impact  are  shown  in  Table  2.3. 

Table  2.3 

Scoring  Criteria  for  Impact 

Score  Criterion 

5  Research  program  has  made  major  contributions  to  worker  health  and  safety  on  the  basis  of 

end  outcomes  or  well-accepted  intermediate  outcomes. 

4  Research  program  has  made  some  contributions  to  end  outcomes  or  well-accepted 

intermediate  outcomes. 

3  Research  program  activities  are  ongoing  and  outputs  are  produced  that  are  likely  to  result  in 

improvements  in  worker  health  and  safety  (with  explanation  of  why  not  rated  higher).  Well- 
accepted  outcomes  have  not  been  recorded. 

2  Research  program  activities  are  ongoing  and  outputs  are  produced  that  may  result  in  new 

knowledge  or  technology,  but  only  limited  application  is  expected.  Well-accepted  outcomes 
have  not  been  recorded. 

1  Research  activities  and  outputs  do  not  result  in  or  are  not  likely  to  have  any  application. 

NA  Impact  cannot  be  assessed;  program  not  mature  enough. 


SOURCE:  National  Academies  (2007,  p.  33). 
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The  criteria  for  assessing  relevance  focused  on  the  EC’s  assessment  of  whether  the 
program  appropriately  set  priorities  among  research  needs  and  the  assessment  of  how 
engaged  the  program  is  in  the  appropriate  transfer  activities.  The  scoring  criteria  for 
relevance  are  shown  in  Table  2.4. 

Table  2.4 

Scoring  Criteria  for  Relevance 


Score  Criterion 


5  Research  is  in  high-priority  subject  areas  and  NIOSH  is  significantly  engaged  in  appropriate 

transfer  activities  for  completed  research  projects  or  reported  research  results. 

4  Research  is  in  priority  subject  areas  and  NIOSH  is  engaged  in  appropriate  transfer  activities 

for  completed  research  projects  or  reported  research  results. 

3  Research  is  in  high-priority  or  priority  subject  areas  but  NIOSH  is  not  engaged  in  appropriate 

transfer  activities,  or  research  focuses  on  lesser  priorities  but  NIOSH  is  engaged  in 
appropriate  transfer  activities. 

2  Research  program  is  focused  on  lesser  priorities  and  NIOSH  is  not  engaged  in  or  planning 

some  appropriate  transfer  activities. 

1  Research  program  is  not  focused  on  priorities  and  NIOSH  is  not  engaged  in  transfer 

activities. 


SOURCE:  National  Academies  (2007,  p.  31). 


RAND's  Role  in  Helping  NIOSH  Programs  Prepare  for  External  Review 

RAND  provided  operational  and  analytical  assistance  to  NIOSH  programs  in  the 
external  review  process.  RAND’s  activities  took  place  in  two  phases:  research  and 
design  (phase  I)  and  implementation  (phase  II).  Each  phase  consisted  of  the  following 
activities: 

•  an  examination  of  the  experiences  of  other  federal  agencies  in  carrying  out  exter¬ 
nal  review;  development  of  a  set  of  guidance  principles  to  assist  NIOSH  programs 
throughout  the  external  review  process;  and  creation  of  a  detailed  set  of  specifica¬ 
tions  for  preparing  a  model  package  of  information  or  “evidence  packages”  to  give 
to  the  reviewers  (Greenfield,  Balakrishnan,  et  ah,  2006) 

•  development  of  a  methodology  to  assist  selected  NIOSH  programs  undergoing 
external  review  and  guidance  on  preparing  evidence  packages.5 


5 


In  Greenfield,  Balakrishnan,  et  al.  (2006),  this  was  referred  to  as  a  submission  package. 
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Phase  I:  Research  and  Design 

In  phase  I,  we  examined  the  experiences  of  other  federal  research  agencies  to  gain 
perspective  about  different  approaches  to  external  review.  Given  a  significant  rime 
constraint,  we  selected  a  small  subset  of  federal  research  agencies  using  the  following 
criteria: 

•  experience  with  external  review,  especially  relating  to  impact6 

•  points  of  commonality  with  NIOSH,  e.g.,  an  application-oriented  perspective 

•  reputation  for  excellence  or  “trendsetting.” 

On  this  basis,  we  developed  a  short  list  of  agencies  that  have  undergone  external 
reviews  of  their  research  programs,  consisting  of  the  U.S.  Environmental  Protection 
Agency  (EPA),  the  U.S.  Department  of  Energy  (DOE),  the  U.S.  Department  of  Agri¬ 
culture  (USDA),  the  National  Institutes  of  Elealth  (NIEI),  and  the  National  Science 
Foundation  (NSF). 

To  complement  the  agencies’  perspectives,  we  also  met  with  staff  of  the  OMB.  In 
contrast  to  the  other  five  agencies,  the  OMB  leads  the  administration’s  PART  process 
and,  in  some  instances,  uses  the  results  of  the  agencies’  external  reviews  as  inputs  to 
that  process.  Moreover,  the  OMB  has  helped  to  shape  the  administration’s  position  on 
external  review. 

Given  our  intent  to  develop  guidance  for  future  reviews,  we  solicited  the  discus¬ 
sants’  views  on  each  of  the  following  topics:  (1)  the  dimensions  of  external  reviews, 
including  program  definitions  and  bases  for  assessment;  (2)  the  internal  management 
of  external  review  processes,  including  the  form  and  extent  of  the  program’s  interac¬ 
tions  with  external  reviewers;  (3)  the  specific  elements  of  the  program’s  evidence  pack¬ 
ages  to  reviewers,  including  program  data  and  data  analysis;  and  (4)  the  results  of 
reviews,  including  both  the  efficiency  and  effectiveness  of  the  process  and  whether  the 
process  yielded  administratively  actionable  and  useful  findings. 

The  results  of  our  analysis  from  phase  I  indicated  four  broad  principles  for  engag¬ 
ing  in  external  reviews  of  impact.  First,  NIOSH  should  define  programs  for  the  best 
possible  fit  with  mission  and  goals;  however,  it  should  also  set  appropriate  expectations 
for  programs  of  different  composition,  size,  and  maturity.  Second,  NIOSH  should 
centralize  coordination  of  the  review  process.  Centralization  will  allow  NIOSH  to 
better  track  the  flow  of  information  to  and  from  reviewers  and  to  establish  institutional 
memory.  Third,  and  most  directly  related  to  the  issue  of  assessing  impact,  NIOSH 
should  use  its  evidence  package  to  tell  the  program’s  research  story.  The  story,  or 
research  narrative ,  should  articulate  answers  to  a  series  of  key  questions,  such  as,  “What 


6  As  discussed  in  Chapter  One,  many  federal  agencies  undertake  external  review,  often  referred  to  as  peer  review , 
to  determine  the  scientific  merit  of  incoming  research  proposals  (e.g.,  responses  to  requests  for  applications, 
requests  for  proposals,  and  program  announcements)  and  to  assess  the  quality  of  research  results  and  outputs 
(such  as  papers  and  methods);  however,  relatively  few  undertake  external  review  to  assess  impact. 
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path  is  the  program  taking  and  why?  What  are  the  program’s  primary  inputs,  activi¬ 
ties,  outputs,  customers,  and  intended  outcomes?  How  does  this  path  support  or  relate 
to  the  program’s  strategy — its  mission  and  goals?  What  are  the  program’s  boundar¬ 
ies  and  responsibilities?  How  does  or  will  research  contribute  to  intended  outcomes?” 
Quantitative  and  qualitative  data  should  be  used  to  validate  the  research  narrative  by 
demonstrating  progress  along  the  path  and  by  establishing  the  research’s  potential  or 
actual  contribution  to  intended  outcomes. 

The  findings  from  this  first  phase  also  indicated  that  logic  models  are  often  used 
to  guide  the  research  narrative.  As  will  be  explained  in  more  detail  in  Chapter  Three,  a 
logic  model  offers  a  simplified  visual  representation  of  the  research  path,  starting  with 
inputs  and  then  progressing  to  activities,  outputs,  customers,  and  intended  outcomes. 
We  identified  at  least  three  roles  for  the  logic  model  in  external  reviews.  First,  it  can 
serve  as  a  communication  device.  It  can  provide  reviewers  with  a  clear  image  or  map 
of  the  program’s  operations  and  intent.  As  such,  it  can  also  provide  a  strong  signal  that 
the  program  understands  its  purpose  and  is  on  track.  Second,  it  can  clearly  identify 
program  boundaries  and  responsibilities,  thereby  clarifying  the  meaning  of  impact  as  it 
relates  to  the  program.  Third,  it  can  provide  a  tool  for  identifying  and  structuring  evi¬ 
dence.  A  program  can  link  qualitative  and  quantitative  indicators  of  progress  to  each 
step  along  the  path,  starting  with  inputs  and  finishing  with  intended  outcomes. 

The  agency  representatives  with  whom  we  met  also  provided  some  specific  recom¬ 
mendations  about  particular  kinds  of  evidence  to  support  the  research  narrative.  They 
said  that  evidence  should  include  planning  documents,  lists  of  clients,  lists  of  facili¬ 
ties  and  offices,  budget  information,  bibliometric  data,  conference  materials,  biblio¬ 
graphies  from  criterion  documents,  research  summaries,  state-of-science  assessments, 
and,  importantly  for  NIOSH,  examples  of  research  contributions  to  intended  out¬ 
comes,  including  client  use  of  research.  Evidence  may  also  take  the  form  of  anecdotal 
narratives,  potentially  supplemented  by  customer  surveys  or  other  forms  of  customer 
feedback  and  external  validation.  In  addition,  submitted  evidence  needs  to  be  shaped 
such  that  the  relevance  and  context  are  apparent.  Simply  handing  over  volumes  of 
information  to  external  reviewers  without  explanation  was  not  considered  to  have  been 
helpful  in  other  reviews. 

Phase  II:  Implementation 

The  second  phase  of  our  project,  the  implementation  phase,  consisted  of  providing 
analytical  and  operational  support  to  seven  of  the  eight  NIOSH  research  programs 
undergoing  review:  Hearing  Loss  Prevention,  Mining,  Respiratory  Diseases,  Trau¬ 
matic  Injury,  Construction,  Personal  Protective  Technology,  and  Health  Hazard  Eval¬ 
uation.  On  average,  we  worked  with  each  program  for  nine  months.  During  this  time, 
we  assisted  in  the  development  of  a  program  logic  model  and  outcome  worksheets  and 
provided  guidance  for  collection  and  presentation  of  evidence  for  the  evidence  pack¬ 
age.  Our  assistance  also  included  the  review  of  initial  and  final  drafts  of  the  evidence 
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package.  Chapters  Three,  Four,  and  Five  of  this  book  describe  the  development  and  use 
of  the  tools  from  the  implementation  phase. 


CHAPTER  THREE 


Logic  Models 


One  of  the  central  challenges  in  demonstrating  impact  is  describing  the  path  by  which 
a  program  achieves  its  end  outcomes.  In  the  case  of  NIOSH  programs,  this  would 
consist  of  a  description  of  how  NIOSH  research  investments,  activities,  and  products 
contribute  to  a  reduction  in  occupational  injuries,  illnesses,  or  fatalities.1  Most  research 
programs  are  a  complex  maze  of  activities  that  include  feedback  loops,  forked  paths, 
and  iterative  cycles.  Simplifying  this  maze  into  a  more  linear  path  is  critical  for  evalu¬ 
ating  the  impact  of  a  research  program.  Thus,  the  starting  point  for  our  implementa¬ 
tion  phase  was  the  development  of  a  program  logic  model  that  served  as  a  blueprint  for 
describing  a  program’s  operational  path.  As  a  blueprint,  the  logic  model  lays  out  the 
program’s  plan  for  how  resources,  activities,  and  outputs  lead  to  outcomes.  However, 
the  logic  model  does  not  claim  to  provide  attribution  of  outcomes  to  activities.  Attri¬ 
bution  involves  firmly  establishing  causal  links  between  observed  changes  and  specific 
interventions,  typically  through  the  use  of  experimental  or  quasi- experimental  research 
designs  that  rely  on  some  controlled  comparison.  These  research  designs  are  often  used 
at  an  individual  activity  or  project  level.  However,  attempting  to  prove  attribution  at 
a  program  level  would  be  a  difficult  undertaking,  given  the  number  of  factors  out¬ 
side  the  program  that  can  influence  the  achievement  of  outcomes.  Thus,  the  program 
logic  model  presents  the  program’s  theory  of  how  research  activities  lead  to  outcomes 
(Wholey,  1983). 

This  chapter  draws  on  our  work  with  NIOSH  programs  to  describe  the  devel¬ 
opment  of  program  logic  models.  In  the  chapter,  we  describe  some  of  the  issues  that 
surfaced  in  trying  to  describe  complicated  research  processes  within  the  boundaries 
of  the  logic  model.  In  so  doing,  we  hope  to  address  concerns  about  the  limitation  of 
logic  models  and  provide  suggestions  and  recommendations  for  representing  contex¬ 
tual  nuances  in  logic  models. 


1  Based  on  John  Mayne’s  (1999)  approach  of  contribution  analysis,  as  a  way  of  measuring  the  performance  of 
public-sector  programs.  This  type  of  analysis  provides  information  on  the  program’s  contribution  to  the  outcomes 
it  is  trying  to  influence  and  uses  a  result  chain  to  illustrate  what  is  supposed  to  happen  as  a  result  of  the  activities 
undertaken  and  the  outputs  produced. 
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Elements  of  a  Logic  Model 

A  logic  model  is  a  visual  depiction  of  how  a  research  program  will  work  under  certain 
environmental  conditions  to  solve  problems  (Bickman,  1987).  As  such,  it  can  be  used 
to  communicate  to  customers  and  other  stakeholders  a  program’s  intended  outcomes 
and  how  resources  and  activities  will  be  used  to  achieve  those  outcomes.  The  logic 
models  that  we  developed  with  the  NIOSH  research  programs  were  based  on  the  stan¬ 
dard  core  elements  described  in  this  section  (see  Figure  3.1). 

Inputs  are  resources  that  both  support  and  guide  program  activities.  Our  logic 
models  distinguish  between  production  resources  (i.e.,  those  resources  that  were  used 
to  support  program  activities,  such  as  funding,  staff,  laboratory  facilities  and  equip¬ 
ment)  and  planning  resources  (i.e.,  those  resources  that  help  to  determine  which  activi¬ 
ties  should  be  undertaken).  Planning  resources  include  strategic  planning  documents, 
surveillance  and  risk-factor  assessment,  legislative  mandates,  and  stakeholder  input. 

Activities  are  the  actions  that  the  program  undertakes  to  produce  outputs.  For 
NIOSH  research  programs,  activities  usually  included  conducting  surveillance;  inves¬ 
tigating  occupational  injuries  and  illnesses;  identifying  risk  and  protective  factors; 
conducting  laboratory  and  field  studies;  developing  and  validating  laboratory  pro¬ 
tocols,  methods,  and  equipment;  assessing  social  and  economic  cost  and  benefit  of 
interventions;  training  and  educating  OSH  professionals;  and  conducting  exposure 
assessments. 

Figure  3.1 

Standard  Elements  for  NIOSH  Program  Logic  Models 
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Outputs  are  the  tangible  products  that  are  generated  by  the  activities.  Outputs 
include  peer-reviewed  journal  articles,  reports  to  Congress,  recommendations,  technol¬ 
ogy,  patents  and  licenses,  and  training  materials. 

Customers  are  defined  as  the  users  or  target  of  the  outputs.  Customers  are  a  criti¬ 
cal  link  in  our  logic  models,  as  they  translate  outputs  to  outcomes.  In  most  research 
programs,  outcomes  cannot  be  achieved  without  customers.  Placing  customers  explic¬ 
itly  in  the  logic  model  helps  program  staff  and  stakeholders  think  through  and  explain 
how  activities  and  outputs  lead  to  outcomes  and  which  population  groups  the  program 
intends  to  serve  (McLaughlin  and  Jordan,  2004).  In  the  NIOSH  logic  models,  we  dis¬ 
tinguish  between  two  types  of  customers:  intermediate  and  final. 

Final  customers  are  the  end  users  and  can  be  thought  of  as  the  target  population 
of  the  research  program.  For  NIOSH  research  programs,  the  final  customers  are  gen¬ 
erally  employers  and  workers.  Employers  may  need  to  implement  changes  at  work  sites 
in  order  to  reduce  injuries,  illnesses,  or  fatalities.  Similarly,  workers  may  need  to  adopt 
recommended  workplace  practices  or  don  particular  workplace  equipment  to  prevent 
injury.  Final  customers  are  key  because,  generally  speaking,  to  achieve  the  end  out¬ 
come,  changes  in  the  behavior  of  the  final  customers  are  necessary. 

Intermediate  customers  are  defined  as  users  or  customers  who  generally  modify  or 
transform  the  outputs  such  that  they  are  more  accessible  to  the  final  customers.  One 
type  of  intermediate  customer  in  our  NIOSH  logic  model  is  a  manufacturing  com¬ 
pany  that  takes  a  pilot  technology  that  NIOSH  has  developed  (an  output)  and  trans¬ 
forms  this  technology  into  market-ready  products  that  are  more  accessible  to  the  final 
customers.  Intermediate  customers  represent  the  reach  of  the  program  and  provide 
the  means  of  moving  outputs  beyond  the  domain  of  the  program  (Montague,  1998). 
Examples  of  intermediate  customers  include  other  U.S.  agencies  (e.g.,  OSHA,  MSHA, 
EPA,  Consumer  Product  Safety  Commission  [CPSC]);  Congress,  state  or  local  OSH 
entities;  international  agencies;  standards-setting  organizations;  labor,  trade,  and  pro¬ 
fessional  associations;  technology  developers;  and  tool,  equipment,  and  material  manu¬ 
facturers.  Although  this  list  is  not  exhaustive,  it  is  important  to  note  that,  in  each  of 
these  examples,  the  role  of  the  intermediate  customer  is  to  extend  the  reach  of  a  par¬ 
ticular  output  through  some  sort  of  modification  that  makes  it  more  accessible  to  the 
final  customer,  whether  in  form  (i.e.,  incorporation  of  NIOSH  recommendations  into 
an  OSHA  regulation)  or  by  proximity  to  a  particular  population  (i.e.,  training  materi¬ 
als  included  in  a  state-based  OSH  training  program). 

Outcomes  are  the  changes  that  occur  and  the  benefits  that  result  from  the  program 
activities  and  outputs.  Some  logic  models  delineate  outcomes  based  on  the  time  frame 
of  when  they  should  occur,  such  as  short-term,  mid-term,  and  long-term  outcomes 
(W.  K.  Kellogg  Foundation,  2000).  Other  logic  models  delineate  outcomes  based  on  a 
hierarchical  system  of  types  of  outcomes,  such  as  the  seven  levels  of  outcomes  described 
by  the  targeted  outcomes  of  programs  (TOP)  model  (Bennett  and  Rockwell,  1995). 
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The  logic  models  we  developed  with  NIOSH  programs  distinguish  between  only  two 
basic  types  of  outcomes:  intermediate  and  end  outcomes. 

End  outcomes  are  the  desired  results  of  the  program.  The  end  outcomes  for  most  of 
the  NIOSH  research  programs  are  reduced  work-related  hazardous  exposures  or  reduc¬ 
tions  in  occupational  injuries,  illnesses,  and  fatalities  within  a  particular  disease-  or 
injury-specific  area.  End  outcomes  describe  the  purpose  of  the  program  and  are  linked 
to  the  program’s  organizational  mission. 

Intermediate  outcomes  represent  the  customers’  response  to  the  program’s  outputs. 
Because  our  logic  models  define  two  types  of  customers — intermediate  and  final — the 
models  also  include  two  types  of  intermediate  outcomes.  The  first  type  of  intermediate 
outcome  is  based  on  the  response  of  the  intermediate  customers  and  is  the  product  that 
has  been  modified,  repackaged,  or  customized  by  the  intermediate  customer  to  make  it 
more  useful  or  accessible  to  the  final  customer.  Using  our  previous  example  of  a  manu¬ 
facturing  company  as  a  NIOSH  intermediate  customer,  the  response  of  transforming 
the  NIOSH  pilot  technology  into  market-ready  products  results  in  the  intermediate 
outcome  of  market-ready  technologies.  These  technologies  are  more  accessible  and  can 
be  directly  used  by  the  final  customers.  The  second  type  of  intermediate  outcome  is 
based  on  the  final  customers’  actions  needed  to  achieve  the  program’s  end  outcome.  For 
example,  if  a  program’s  end  outcome  were  reduced  occupational  injuries  at  construction 
sites,  then  an  intermediate  outcome  would  be  an  employer’s  (final  customer’s)  adop¬ 
tion  of  new  technologies  at  the  workplace  that  makes  the  construction  site  safer.  There 
are  several  of  these  types  of  intermediate  outcomes  that  were  common  to  the  NIOSH 
research  programs.  These  outcomes  included  (1)  adoption  of  technologies;2  (2)  changes 
in  workplace  policies,  practices,  and  procedures;3  (3)  changes  in  the  physical  environ¬ 
ment  (i.e.,  working  environment)  and  organization  of  work;  and  (4)  changes  in  knowl¬ 
edge,  attitudes,  and  behavior  of  the  final  customers  (i.e.,  employees,  employers). 

External factors  are  also  included  in  our  logic  models  to  acknowledge  aspects  that 
are  outside  of  the  research  program  yet  influence  (either  positively  or  negatively)  the 
extent  to  which  a  program  can  achieve  its  end  outcomes.  For  research  programs  within 
NIOSH,  these  external  factors  could  include  industry-specific  legislation,  major  inci¬ 
dents  and  disasters  (i.e.,  mining  explosion  disasters),  the  political  environment,  techno¬ 
logical  developments,  and  market  forces.  Figure  3.1  shows  a  series  of  arrows  pointing 
upward  from  external  factors  to  indicate  that  these  factors  are  linked  to  and  can  influ¬ 
ence  different  elements  within  the  logic  model.  For  example,  external  factors,  such  as 
access  to  various  databases,  may  affect  program  activities,  such  as  surveillance.  On  the 


2  An  example  of  an  adoption  of  a  new  technology  is  computer  software  developed  by  the  Mining  program  to 
analyze  pillar  stability.  This  is  now  an  industry  standard  and  is  used  by  MSHA  and  state  regulatory  agencies 
to  evaluate  mine  permits,  resulting  in  safer  longwall  operations. 

3  An  example  of  changes  in  workplace  policies  is  the  passage  of  safe  patient-handling  legislation  in  six  states 
from  2005  to  2006.  This  legislation  was  based  on  patient-handling  research  conducted  by  the  Traumatic  Injury 
program. 
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other  hand,  external  factors,  such  as  government  earmarks,  would  influence  program 
inputs. 


Importance  of  Logic  Models  in  Demonstrating  and  Assessing  Impacts 

As  described  in  Chapter  One,  there  are  many  methodological  hurdles  to  evaluating 
and  demonstrating  the  impact  of  research  programs.  Logic  models  offer  a  means  of 
addressing  some  of  them.  For  example,  the  links  between  research  activities  and  out¬ 
comes  are  often  complex  and  long-term.  Models  that  can  articulate  an  observable  and 
measurable  path  from  inputs,  activities,  and  outputs  through  a  series  of  intermedi¬ 
ate  and  end  outcomes  can  help  determine  whether  the  research  activities  are  more  or 
less  likely  to  achieve  the  desired  societal  objectives  or  impacts  (Roessner,  2002).  Logic 
models  also  help  to  define  the  domain  of  analysis  for  evaluating  impact.  For  many 
research  evaluations,  it  is  difficult  to  move  beyond  assessing  the  merit  and  work  of 
individual  research  projects.  Logic  models  provide  a  comprehensive  view  of  a  research 
program — from  what  it  does  to  whom  it  affects  and  the  expected  outcomes — that 
can  form  the  basis  of  the  evaluation.  As  described  in  Chapter  Two,  the  Framework 
Committee  created  its  evaluation  flow  chart  from  the  NIOSH  logic  model  (see  Appen¬ 
dix  A).  By  segmenting  the  NIOSH  logic  model  into  discrete,  sequential  program  com¬ 
ponents,  the  Framework  Committee  created  a  flow  chart  that  summarizes  how  the 
program  evaluation  should  occur. 

Finally,  because  of  the  multiple  contributors  to  any  given  end  outcome,  the  logic 
model  helps  define  the  program’s  sphere  of  influence — i.e.,  for  what  the  program  can 
take  credit  and  be  held  responsible.  Figure  3.2  illustrates  how  the  logic  model  distin¬ 
guishes  program  efforts  from  program  effects  and  thus  defines  where  a  program  is 
likely  to  have  direct  control,  direct  influence,  and  indirect  influence. 

The  program  has  direct  control  over  its  inputs,  activities,  and  outputs  (except  for 
the  influence  of  external  factors).  Once  the  program  transfers  its  outputs  to  its  custom¬ 
ers,  the  program  may  have  less  control  but  may  influence  how  its  customers  use  its 

Figure  3.2 
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outputs.  At  the  outcome  level,  a  program  has  indirect  influence  at  best,  but  it  continues 
to  bear  responsibility  for  contributing  to  its  intended  outcomes. 


Linking  Program  Operations  to  Program  Strategy 

One  useful  feature  of  this  model  is  the  link  between  standard  elements  (i.e.,  inputs,  activ¬ 
ities,  outputs,  and  outcomes)  and  the  mission  and  goals  of  the  program  (Figure  3.3). 

We  refer  to  the  standard  elements  as  depicting  the  program  operations  and  the 
program  goals  (i.e.,  strategic,  intermediate,  and  annual)  as  depicting  the  program  strat¬ 
egy.  Note  that  program  operations  flow  from  left  to  right  and  include  the  standard  logic 
model  elements  described  in  the  previous  section. 

In  contrast,  program  strategy  flows  from  right  to  left  and  includes  the  following: 
Strategic  goals  (such  as  a  reduction  in  the  incidence  of  a  particular  type  of  injury) 
derive  from  the  program’s  mission;  intermediate  goals  (such  as  an  increase  in  public 
knowledge  regarding  proper  safety  measures)  derive  from  strategic  goals;  annual  goals 
(such  as  the  conduct  of  research  on  proper  safety  measures  and  the  publication  of  safety 
reports)  derive  from  intermediate  goals;  and  management  objectives  (such  as  those 
regarding  the  efficient  use  of  NIOSH  resources)  derive  from  preceding  goals. 

Connecting  these  two  fundamental  components  of  a  research  program — program 
operations  and  program  strategy  via  the  program  mission — provides  an  internal  check 
that  what  the  program  is  doing  and  the  goals  it  has  defined  for  itself  are  vertically 
aligned.  This  alignment  allows  the  logic  model  to  be  used  as  a  program  management 

Figure  3.3 
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tool  because  it  provides  structure  and  guidance  for  how  a  program  should  define  its 
strategic,  intermediate,  and  annual  goals.  Generally  speaking,  strategic  goals  should 
relate  to  the  program’s  contribution  to  end  outcomes;  intermediate  goals  should  relate 
to  customers  and  intermediate  outcomes,  such  as  those  involving  changes  in  policies, 
behavior,  or  the  work  environment;  annual  goals  should  relate  to  program  activities 
and  outputs;  and  management  objectives  should  relate  to  program  inputs  and  activi¬ 
ties.  We  refer  to  this  linking  of  program  operations  to  program  strategy  as  the  logic 
model  horseshoe  template  and  use  this  template  in  the  logic  models  we  developed 
(Greenfield,  Williams,  and  Eiseman,  2006). 


Generating  the  Information  for  the  Logic  Model 

Filling  in  the  logic  models  with  specific  information  about  a  research  program  requires 
detailed  information  about  research  program  operations.  Surprisingly,  this  informa¬ 
tion  is  not  often  readily  available  and  is  best  generated  through  discussions  with  pro¬ 
gram  personnel.  Because  no  single  individual  can  provide  all  of  the  information  nec¬ 
essary  to  explicate  the  complete  program-operation  path,  meetings  with  key  program 
personnel,  including  program  managers,  coordinators,  and  staff  scientists,  are  neces¬ 
sary  to  build  a  logic  model.  Next  is  a  set  of  discussion  points  that  we  used  in  meet¬ 
ings  with  NIOSH  program  staff  to  facilitate  the  development  of  the  logic  model.  The 
discussion  points  were  intended  to  elicit  information  about  program  operations  and 
strategy  and  to  closely  parallel  the  structure  of  the  stylized  logic  model  shown  in  Fig¬ 
ures  3.1  and  3.3.  In  abbreviated  form,  we  addressed  the  following  questions: 

•  What  is  your  program  trying  to  achieve  and  why? 

•  Who  are  your  customers,  partners,  and  other  stakeholders? 

•  What  types  of  infrastructure-support  activities  (e.g.,  planning  and  funding  pro¬ 
cesses,  laboratories)  does  your  program  undertake? 

•  What  other  inputs  are  used  to  generate  activities? 

•  What  does  the  program  do  (e.g.,  conducts  surveillance,  undertakes  or  funds 
research,  develops  and  evaluates  interventions)? 

•  Do  you  work  with  partners?  If  so,  who  are  they  and  how  do  you  work  with 
them? 

•  What  does  the  program  produce  (e.g.,  papers,  methods,  technologies,  training  or 
educational  materials,  workshops,  programs)? 

•  How  are  the  outputs  disseminated  or  transferred  to  others? 

•  How  (and  by  whom)  are  the  outputs  used  and  for  what  purposes? 

•  What  are  the  intended  outcomes  of  your  activities? 

•  What  are  the  program’s  boundaries  and  niche? 

•  What  external  factors  affect  your  efforts? 
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The  information  gathered  from  the  discussions  was  used  to  develop  an  initial 
draft  of  the  logic  model.  The  first  step  was  to  categorize  the  information  gathered 
from  discussion  with  program  staff  within  the  correct  logic  model  element.  In  most 
instances,  this  categorization  was  fairly  straightforward.  However,  in  other  cases,  con¬ 
textual  details  were  critical  in  determining  the  correct  categorization.  For  example, 
distinguishing  an  output  from  an  outcome  hinges  on  the  involvement  of  customers, 
so  understanding  who  is  doing  what  at  each  stage  of  the  logic  model  is  critical.  Once 
the  text  was  categorized  correctly,  the  next  step  was  determining  which  information 
needed  to  be  featured  in  the  logic  model.  In  general,  the  information  collected  from 
discussions  with  program  personnel  described  individual  projects  within  a  program. 
However,  for  the  logic  model,  the  research  program — not  a  project — was  the  appro¬ 
priate  unit  of  analysis.  Thus,  to  represent  the  totality  of  the  research  program,  we 
combined  similar  inputs,  activities,  outputs,  and  outcomes.  Representing  research  pro¬ 
grams  as  logic  models  depended  on  using  concise  terminology  while  maintaining  as 
much  detail  as  possible  to  relay  the  complexity  of  a  program’s  operations.  We  paid  par¬ 
ticular  attention  to  the  language  used  to  describe  each  element  of  the  logic  model.  For 
activities,  we  consistently  used  verbs  to  describe  what  was  done  (e.g.,  develop  guidance 
documents,  develop  and  validate  model  investigation  protocols  and  analytic  methods),  and, 
for  outputs,  we  consistently  used  nouns  to  describe  what  was  produced  (e.g.,  guidance 
documents,  model  protocols,  new  analytic  methods).  Since  many  of  the  types  of  inputs, 
activities,  outputs,  customers,  and  outcomes  are  similar  between  the  various  NIOSH 
programs,  we  maintained  as  much  consistency  as  possible  in  the  terminology  used  for 
each  program’s  logic  model. 

One  of  the  more  rigorous  aspects  of  the  logic  models  was  the  attention  we  paid  to 
connecting  the  information  within  the  boxes  across  each  of  the  logic  model  elements. 
For  example,  we  made  sure  that  each  output  was  linked  to  one  or  several  activities,  and 
each  intermediate  outcome  was  linked  back  to  specific  outputs  that  a  program’s  cus¬ 
tomers  used  or  adopted.  An  example  of  this  linking  is  shown  in  Figure  3.4. 

The  program  activity  train  and  educate  OSH  professionals  results  in  the  outputs 
of  trained  OSH  professionals  and  training  and  educational  materials.  These  training  and 
educational  materials  may  be  transferred  to  intermediate  customers  who  use  these  mate¬ 
rials  to  develop  training  programs,  an  intermediate  outcome  of  the  research  program. 
Final  customers,  such  as  employers  and  employees,  may  participate  in  these  training 
programs,  leading  to  changes  in  workplace  practices,  another  category  of  intermedi¬ 
ate  outcomes.  Over  time,  these  changes  in  workplace  practices  could  contribute  to  a 
reduction  in  occupational  injuries,  the  end  outcome. 

Program  staff  provided  input  on  each  iteration  of  the  logic  model,  specifically 
on  its  accuracy,  content,  structure,  and  flow.  It  was  during  these  iterations  that  we 
developed  some  of  the  nuances  that  are  present  in  the  final  logic  models.  For  exam¬ 
ple,  the  difference  between  resources  used  to  support  activities  and  resources  used  to 
determine  activities  to  be  undertaken  was  considered  to  be  an  important  distinction 
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Figure  3.4 

Linking  Elements  Within  Logic  Model  Boxes 
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among  NIOSH  research  programs  and  thus  production  and  planning  resources  became 
a  standard  feature  of  the  logic  models  we  generated.  The  last  stage  of  our  logic  model 
development  involved  adding  details  to  the  model  that  established  some  distinctive 
elements  of  each  of  the  research  programs,  such  as  the  depiction  of  a  program’s  rela¬ 
tionship  with  its  partners,  and  the  incorporation  of  information  about  programs  with 
both  research  and  service  components.  Working  with  program  staff,  the  logic  model 
was  determined  to  be  complete  when  there  was  general  consensus  among  all  involved 
about  the  accuracy  of  the  logic  model’s  representation  of  the  research  program.  In  the 
next  section,  we  discuss  some  of  the  nonstandard  elements  of  the  logic  models  that  we 
developed  with  NIOSH  programs. 


Approaches  for  Addressing  Research  Complexity  in  the  Logic  Models 

Despite  the  standard  structure  (i.e.,  boxes  and  arrows)  and  elements  (i.e.,  inputs,  activi¬ 
ties,  outputs,  and  outcomes)  in  logic  models,  there  are  a  number  of  ways  to  customize 
these  models  so  that  details  that  are  critical  to  illustrating  program  operations  can  be 
included.  The  use  of  various  formatting  tools,  such  as  italics,  bolded  text,  dashed  lines, 
and  colors,  can  be  used  to  indicate  subtle  differences.  However,  finding  the  balance 
between  the  simplicity  of  the  basic  flow  of  the  logic  model  and  including  important 
contextual  features  is  challenging.  A  logic  model  is,  by  necessity,  an  abstraction  that 
omits  detail  for  the  sake  of  clarity  of  representation.  The  trade-off  between  detailed 
accuracy  and  clarity  of  presentation  varies  according  to  context  and  purpose.  In  devel¬ 
oping  logic  models  with  NIOSH  programs,  clearly  depicting  the  paths  to  outcomes 
was  the  priority.  Thus,  we  maintained  the  linear  flow  of  the  logic  model,  using  feed¬ 
back  loops  and  other  divergent  pathways  sparingly.  Because  the  programs  were  under¬ 
going  external  review,  we  favored  a  more  simplified  model  that  would  enable  a  reviewer 
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to  look  at  the  logic  model  and  see  the  path  of  how  a  program  achieves  its  end  outcomes. 
Although  we  recognized  that  there  might  be  exceptions,  the  expert  reviewers’  ability  to 
assess  impact  depended,  in  part,  on  their  understanding  of  how  the  program  operated. 
We  did  not  want  to  jeopardize  the  clarity  of  the  model  by  including  too  many  details. 
Moreover,  because  these  logic  models  were  included  as  part  of  the  evidence  package, 
many  of  the  details  not  shown  in  the  models  were  explained  in  different  sections  of  the 
package. 

In  developing  logic  models  with  the  different  research  programs,  we  encountered 
a  number  of  complexities  that  necessitated  varying  the  logic  models.  Two  of  the  most 
significant  ones  were  representing  partnerships  in  logic  models  and  describing  research 
programs  that  did  more  than  just  research.  Both  of  these  conditions  are  conceptually 
complex  and  common  to  many  research  programs.  In  the  following  sections,  we  dis¬ 
cuss  some  of  the  challenges  we  faced  in  addressing  these  issues  and  the  approaches  we 
used  to  resolve  them. 

Representing  the  Role  of  Partners  in  Logic  Models:  The  Mining  and  Construction 
Research  Programs 

NIOSH  programs  collaborate  or  partner  with  many  other  federal,  state,  local,  and  pri¬ 
vate  organizations,  including  federal  agencies  (e.g.,  OSHA,  Bureau  of  Labor  Statistics 
[BLS],  CDC);  state  health  and  labor  departments;  standards-development  organiza¬ 
tions  (e.g.,  International  Organization  for  Standardization  [ISO],  American  National 
Standards  Institute  [ANSI]);  labor,  trade,  and  professional  associations;  and  technol¬ 
ogy  developers  and  manufacturers.  Thus,  depicting  the  contribution  of  NIOSH  part¬ 
ners  in  achieving  outcomes  was  an  important  element  to  add  to  the  logic  models. 

Representing  partners  in  the  logic  models  was  complicated  by  trying  to  determine 
how  they  differed  from  customers,  which  were  already  present  in  the  logic  models.  For 
many  of  the  programs  with  which  we  worked,  partners  primarily  played  a  supportive 
role,  enabling  the  research  programs  to  carry  out  activities  or  produce  outputs.  Thus, 
we  defined  partners  as  those  who  work  with  research  programs  to  conduct  activities 
or  enable  outputs,  whereas  customers  are  defined  as  the  users  or  targets  of  the  outputs. 
Note  that  these  definitions  are  not  mutually  exclusive,  and  the  same  entity  that  serves 
as  a  partner  in  one  context  may  serve  as  a  customer  in  another.  For  example,  state 
health  and  labor  departments  may  work  with  NIOSH  to  produce  training  and  educa¬ 
tion  programs  and  can  therefore  be  described  as  partners;  however,  they  can  also  use 
these  outputs  as  customers. 

To  represent  partners  and  the  supportive  role  (rather  than  a  more  direct  or  active 
role),  we  used  a  shaded  box  to  encompass  the  logic  model  elements  in  which  partners 
played  a  key  role.  At  the  bottom  of  the  box,  we  listed  the  partners  that  the  research 
programs  described  for  us.  Although  this  represented  our  general  strategy  for  depicting 
partners,  there  was  some  variation  in  which  boxes  were  encompassed  and  which  groups 
or  entities  were  included  as  partners.  Clearly,  the  full  complexity  of  partners  and  their 
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relationship  with  NIOSH  programs  is  not  accounted  for  by  simply  adding  another  box 
to  the  logic  model.  However,  this  additional  component  (i.e.,  the  green  shaded  box) 
was  used  to  underscore  the  importance  of  partnership  in  the  program’s  path  to  achiev¬ 
ing  end  outcomes. 

In  the  following  sections,  we  describe  logic  models  for  two  of  the  NIOSH  pro¬ 
grams  we  assisted:  Construction  and  Mining.  The  role  of  partners  in  these  two  research 
programs  is  different,  and  we  discuss  how  we  illustrated  these  differences  in  the  logic 
models. 

The  Construction  Program.  The  Construction  program  was  created  in  1990  by 
Congress  to  “develop  a  comprehensive  prevention  program  directed  at  health  prob¬ 
lems  affecting  construction  workers  by  expanding  existing  NIOSH  activities  in  areas 
of  surveillance,  research  and  intervention”  (NIOSH,  2007a).  The  program  is  made  up 
of  three  main  components:  NIOSH-wide  intramural  research  and  surveillance  pro¬ 
grams;  a  large  national  Construction  Center  cooperative  agreement;  and  investigator- 
initiated  extramural  research.  Partnerships  are  integral  to  the  Construction  program. 
Research  with  its  partners  sometimes  includes  in-kind  contributions  that  help  to  lever¬ 
age  NIOSH  research  dollars  (NIOSH,  2007a).  Partners  can  also  add  expertise  or  spe¬ 
cialized  experience  to  the  research  team.  The  Construction  program  has  partnerships 
with  several  key  players  involved  with  construction  issues,  including  OSHA;  the  con¬ 
struction  industry;  labor,  trade,  and  professional  organizations  (e.g.,  United  Brother¬ 
hood  of  Carpenters  and  Joiners  of  America  [UBC];  Laborers’  International  Union  of 
North  America  [LIUNA]);  and  academic  institutions  (e.g.,  NIOSH  Education  and 
Research  Centers  [ERCs]). 

One  partnership  in  particular  is  integral  to  the  Construction  program — its  part¬ 
nership  with  the  Center  for  Construction  Research  and  Training,  formerly  known  as 
the  Center  to  Protect  Workers’  Rights  (CPWR),  a  research  center  established  by  the 
Building  and  Construction  Trades  Department  of  the  American  Federation  of  Labor 
and  Congress  of  Industrial  Organizations  (AFL-CIO)  that  includes  a  consortium 
of  academic  and  industry  research  organizations.  Since  1996,  the  CPWR  has  been 
awarded  the  national  Construction  Center  cooperative  agreement,  which  is  intended  to 
promote  dialogue  and  collaboration  among  researchers.  As  a  partner,  the  Construction 
Center  provides  important  linkages  to  the  construction  community  and  focuses  and 
coordinates  research  that  is  often  more  applied  than  NIOSH  intramural  research. 

The  Construction  program  uses  the  term  partners  to  refer  to  stakeholder  groups 
that  collaborate  with  it  on  program  activities.  In  the  logic  model  we  developed  with 
the  Construction  program,  we  depict  its  partnerships  as  enabling  both  activities  and 
outputs  by  drawing  the  shaded  partnership  box  around  the  activities  and  outputs  boxes 
(Figure  3.5). 

To  denote  the  dual  role  of  partners  as  customers,  many  of  the  entities  listed  as 
partners  are  also  in  the  “Intermediate  customers”  box.  Having  partners  that  also  were 
intermediate  customers  was  a  common  strategic  decision  by  many  of  the  research  pro- 
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Figure  3.5 

Logic  Model  for  the  NIOSH  Construction  Program 
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published  documents 
(e.g.,  alerts,  guidance 
documents,  HHE  and 
FACE  reports);  other 
publications  (e.g., 
reports  and  monographs 
in  trade  journals,  con¬ 
ference  proceedings); 
reports  to  Congress; 
authoritative  recom¬ 
mendations  (e.g., 
exposure  limits);  best 
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tools,  methods,  and 
equipment;  conferences; 
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trained  OSH 
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Partnerships  with  other  NIOSH  program  areas;  other  U.S.  agencies 
(e.g.,  OSHA,  BLS);  state  health  and  labor  departments;  local  agencies; 

international  agencies;  NGOs;  consensus  standards-setting 
organizations;  labor,  trade,  and  professional  associations  (e.g.,  CPWR); 
technology  developers;  tool,  equipment,  and  material  manufacturers; 
construction  industry  (e.g.,  contractors,  building  owners);  joint 
labor-management  associations;  apprenticeship  trainee  programs; 
insurers;  academic  institutions  (e.g.,  ERCs);  and  others 
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reports  and 
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SOURCE:  NIOSH  (2007). 

NOTE:  CFOI  =  Census  of  Fatal  Occupational  Injuries.  FACE  =  Fatality  Assessment  and  Control  Evaluation.  IMIS  =  Integrated  Management  Information 
System.  r2p  =  research  to  practice.  CSC  =  Construction  Steering  Committee.  PPE  =  personal  protective  equipment.  eLCOSH  =  Electronic  Library  of 
Construction  Occupational  Safety  and  Health.  NGO  =  nongovernmental  organization. 

a  NIOSH  laboratories  and  other  facilities  used  to  conduct  research  through  grants,  cooperative  agreements,  and  contracts. 
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grams  with  which  we  worked.  Involving  intermediate  customers  early  on  as  partners 
helped  to  ensure  that  the  outputs  that  customers  received  were  relevant  and  increased 
the  likelihood  that  customers  would  use  the  outputs  resulting  in  the  expected  program 
outcomes. 

The  Mining  Program.  The  Mining  program  partnerships  facilitate  advances  in  the 
safety  and  health  of  U.S.  mine  workers  through  input  from  customers  and  stakeholder 
groups  to  help  set  research  priorities,  in-kind  contributions  (such  as  equipment  and  test 
mine  sites)  to  extend  research  dollars,  and  expertise  or  specialized  experience  to  supple¬ 
ment  the  research  team.  The  Mining  program  groups  its  partnerships  into  three  cat¬ 
egories:  those  with  (1)  important  customers  and  stakeholders  to  identify  research  needs 
and  transfer  research  findings,  (2)  manufacturers  and  mining  companies  to  conduct 
research  and  develop  products,  and  (3)  organizations  to  exchange  safety  and  health 
information  and  technology  and  to  conduct  research  in  areas  of  mutual  benefit.  These 
partnerships  enhance  the  Mining  program  and  also  allow  for  faster  transfer  of  knowl¬ 
edge  and  products  to  the  mining  industry. 

In  comparison  with  the  Construction  program,  the  Mining  program  has  a  more 
complicated  relationship  with  its  partners.  The  Mining  program  works  very  closely 
with  a  broad  spectrum  of  members  of  the  mining  community,  and,  similar  to  Con¬ 
struction,  many  of  the  partners  are  also  stakeholders  and  intermediate  customers.  How¬ 
ever,  Mining  differs  in  that  there  is  considerable  overlap  in  the  ways  in  which  these 
different  groups  contribute  to  outcomes.  In  particular,  the  Mining  program  considers 
its  relationships  with  partners,  stakeholders,  and  intermediate  customers  integral  to 
the  transfer  of  their  research  findings.  Therefore,  partners,  stakeholders,  and  custom¬ 
ers  were  grouped  together  in  a  shaded  box  encompassing  activities,  outputs,  transfer 
activities,  and  the  first  intermediate  outcomes  box  (Figure  3.6).  The  shaded  box  lists 
some  of  the  primary  partners,  stakeholders,  and  intermediate  customers  of  the  NIOSH 
Mining  program. 

The  Mining  program  logic  model  suggests  a  close  relationship  with  partners, 
stakeholders,  and  customers  that  essentially  extends  the  sphere  of  direct  control.  In 
contrast  to  the  Construction  program,  in  which  there  is  a  clear  transfer  of  outputs  to 
intermediate  customers,  there  is  no  transfer  arrow  to  intermediate  or  final  custom¬ 
ers  in  the  Mining  Program  logic  model.  The  Mining  program  engages  with  research 
partners,  stakeholders,  and  customers  through  to  intermediate  outcomes  and,  conse¬ 
quently,  assumes  greater  responsibility  for  achieving  intermediate  outcomes.  This  is 
illustrated  in  Figure  3.6,  in  which  the  sphere  of  direct  control  in  the  Mining  research 
program  ranges  from  inputs  to  the  first  “Intermediate  outcomes”  box. 

Describing  Multifaceted  Research  Programs:  The  Health  Hazard  Evaluation  and 
Personal  Protective  Technology  Programs 

Two  NIOSH  programs  presented  the  challenge  of  how  to  represent  those  that  do  more 
than  just  research.  The  HHE  program  has  a  service  component  of  responding  to  cus- 
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Logic  Model  for  the  NIOSH  Mining  Program 


Mission:  To  eliminate  occupational  diseases,  injuries,  and  fatalities  from  the  mining  workplace  through  a  focused  program  of 

research  and  prevention 


Transfer  activities 


international  agencies;  NGOs;  labor,  trade,  and  professional  associations;  technology 
developers  and  manufacturers;  researchers  in  science  and  technology 

t _ 1 _ 1 _ i _ 1 _ i _ EXternalfaCt°rS _ i _ i _ 1 _ i _ 1 _ t 

V  Y  A  Y  A  Y  J 

Sphere  of  Sphere  of  direct  Sphere  of 

direct  control  influence  indirect  influence 

SOURCE:  NIOSH  (undated  [b]). 

NOTE:  The  mining  program  created  its  logic  model,  so  the  headings  and  overall  structure  of  this  logic  model  differ  slightly  from  the  others  that 
we  illustrate. 
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tomer  requests  for  hazard  evaluations.  The  PPT  program  has  a  component  that  is 
responsible  for  certifying  all  respirators  used  in  the  United  States.  In  both  cases,  addi¬ 
tional  details  were  included  in  the  logic  models  to  create  a  more  accurate  depiction  of 
the  programs  and  how  they  achieve  their  outcomes. 

The  Health  Hazard  Evaluation  Program.  The  HHE  program  is  designed  to 
respond  to  requests  for  field  investigations  of  potential  health  hazards  in  the  workplace. 
These  field  investigations  are  called  health  hazard  evaluations,  or  HHEs.  However, 
there  is  also  a  research  component  to  this  program,  as  some  of  these  requests  can  pres¬ 
ent  opportunities  for  further  investigations.  Figure  3.7  is  the  logic  model  we  developed 
in  collaboration  with  the  HHE  program. 

The  service  component  of  the  HHE  program  is  represented  in  the  path  on  which 
the  program’s  outputs  are  transferred  directly  to  its  final  customers:  (1)  employees, 
managers,  and  employers  at  investigated  and  other  facilities  and  (2)  labor  unions.  In 
this  case,  a  feedback  loop  from  the  final  customers  to  the  inputs  is  used  to  indicate 
that  customer  feedback  from  these  investigations  informs  production  inputs  as  well  as 
planning  inputs  for  future  program  activities.  The  research  component  of  the  HHE 
program  is  represented  in  the  path  on  which  the  outputs  are  transferred  to  intermedi¬ 
ate  customers  who  modify  and  repackage  the  outputs  so  they  are  more  accessible  to 
final  customers. 

The  HHE  program  logic  model  also  illustrates  another  feature  that  we  incorpo¬ 
rated  into  some  of  our  logic  models:  the  expanded  “Transfer”  arrow.  The  Framework 
Committee  considered  transfer  activities  to  be  distinct  from  research  activities.  For 
programs,  such  as  HHE,  that  used  different  types  of  transfer  mechanisms  to  relay  its 
outputs  to  the  appropriate  customers,  these  mechanisms  are  specifically  detailed  in  the 
“Transfer”  arrow  in  the  logic  model. 

The  Personal  Protective  Technology  Program.  The  mission  of  the  PPT  program 
is  to  prevent  work-related  injury,  illness,  and  death  by  advancing  the  state  of  knowl¬ 
edge  and  application  of  personal  protective  technologies.  Although  the  PPT  program 
was  not  established  until  2005,  NIOSH  had  initiated  a  research  program  on  personal 
protective  technologies  as  early  as  1973.  The  PPT  program  fulfills  its  mission  through 
three  major  areas:  (1)  research,  (2)  participation  in  standards  setting  and  policymak¬ 
ing,  and  (3)  respirator  certification  (NRC,  2008).  The  complexity  introduced  by  these 
three  distinct  areas  is  evident  in  the  logic  model  shown  in  Figure  3.8. 

The  respiratory  protection  research  conducted  by  the  PPT  program  and  the  NIOSH 
respirator-certification  program  follow  two  parallel  paths  within  the  PPT  program. 
Both  paths  are  guided  by  the  same  inputs,  but  the  congressional  mandate  to  conduct 
respirator  certification  dictates  that  the  emphasis  on  the  respective  inputs  be  different 
for  each  path  (NIOSH,  2007b).  Respiratory  research  and  the  respirator-certification 
program  each  have  their  own  set  of  activities,  outputs,  intermediate  customers,  and 
intermediate  outcomes.  There  are  some  interactions  between  the  research  path  and  the 
respirator-certification  path,  indicated  by  arrows  connecting  the  two  paths.  Some  of 
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Figure  3.7 

Logic  Model  for  the  NIOSH  Health  Hazard  Evaluation  Program 
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review  processes 

Planning: 

HHE  requests;  triage 
process  and  procedure 
manual;  follow-back 
data;  strategic-planning 
documents  (e.g.,  NORA, 
r2p);  legislative  man¬ 
dates  (e.g.,  OSHAct, 
MSHAct,  42  CFR85); 
NIOSH  priorities;  prior 
HHE  program  reviews; 
NIOSH  Human  Subjects 
Review  Board  approval; 
stakeholder  input; 
scientific  knowledge 
base 


Describe  and  document  work¬ 
place  conditions,  exposures, 
and  employee  health  effects; 
assess  relationships  between 
workplace  exposures,  conditions, 
and  employee  health;  evaluate 
effectiveness  of  exposure  con¬ 
trols  and  other  workplace  inter¬ 
ventions;  develop  and  validate 
model  investigation  protocols 
and  analytic  methods;  recom¬ 
mend  solutions  to  reduce  work- 
related  health  hazards;  educate 
employers  and  employees  about 
health  hazards;  make  referrals  to 
OSH  professionals  and  other 
public-health  agencies;  respond 
to  general  inquiries  for  informa¬ 
tion  and  consultation;  provide 
technical  assistance  to  OSH  pro¬ 
fessionals  and  other  public-health 
agencies;  develop  guidance 
documents;  participate  in  pro¬ 
fessional  and  agency  commit¬ 
tees  and  work  groups;  review 
technical  and  policy  documents; 
train  OSH  professionals 


HHE  reports;  NIOSH  pub¬ 
lished  documents  (e.g., 
numbered  documents, 
alerts,  topic  pages, 
summaries  of  HHEs); 
peer-reviewed  journal 
articles;  other  publica¬ 
tions  (e.g.,  in  other 
agencies'  documents, 
trade  journals,confer- 
ence  proceedings); 
testimony,  recommen¬ 
dations,  best  practices, 
technology,  patents 
and  licenses,  new 
analytical  methods, 
model  protocols 
(e.g.,  field 
investigations, 
laboratory 
analyses),  meeting 
presentations, 
educational  and 
training  materials, 
trained  OSH 
professionals,  Web 
sites  (e.g.,  NIOSH, 
CDC) 


Partnerships  with  other  NIOSH  program  areas,  other  U.S. 
agencies  (e.g.,  OSHA,  CDC),  state  and  local  health 
departments,  contract  laboratories,  and  others 


Transfer:  announce  HHE  investigations  on 
Epi-X;*3  send  HHE  reports  to  employers,  employees, 
and  OSHA;  post  HHE  reports  on  Web  sites, 
announce  on  eNews,  and  publish  in  MMWR; 
present  HHE  findings  at  scientific  and  trade 
meetings;  publish  HHE  findings  in  scientific  and 
trade  publications;  issue  NIOSH  alerts 


Intermediate 

customers: 

other  NIOSH  program 
areas;  other  U.S. 
agencies  (e.g.,  OSHA, 
CDC);  Congress;  state 
and  local  entities;  inter¬ 
national  agencies; 
NGOs;  labor,  trade, 
and  professional 
associations;  standards- 
development  organi¬ 
zations;  technology 
manufacturers;  OSH 
and  medical  profes¬ 
sionals;  academic  insti¬ 
tutions;  researchers; 
lawyers;  media 


Guidelines, 
policies,  standards, 
regulations,  and  laws; 
litigation;  new  research 
activities  and  revised 
scientific  agendas; 
commercial 
analytic  and 
control  products; 
training  and 
educational  programs; 
expanded  OSH 
workforce;  revised 
public  and  media 
agendas;  media 
releases; 

committee  reports; 
Web  sites 


Final 

customers: 

employees, 
managers,  and 
employers  at 
investigated 
facilities; 
labor  unions 


Employees, 
managers, and 
employers  at 
non-investigated 
facilities; 
labor  unions 


Changes  in 
workplace 
policies, 
practices, 
and 

procedures; 
changes  in 
physical 
environment 
and  organization 
of  work; 
changes  in 
knowledge, 
attitudes,  and 
behavior 


Reduction  in 
occupational 
illnesses  and 
exposure  to 
workplace 
hazards 


Management  objectives 

Annual  goals 

◄- 

Intermediate  goals 

Strategic 

goals 
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SOURCE:  NIOSH  (undated  [c]). 

NOTE:  HETAB  =  Hazard  Evaluations  and  Technical  Assistance  Branch.  FSB  =  Field  Studies  Branch.  MMWR  =  Morbidity  and  Mortality  Weekly  Report. 
NIOSH  laboratories  and  other  facilities  accessed  through  contracts. 

Epi-X  is  the  Epidemic  Information  Exchange,  CDC's  secure  communication  network  for  public-health  professionals. 
c  eNews  is  NIOSH's  monthly  newsletter. 
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Figure  3.8 

Logic  Model  for  the  NIOSH  Personal  Protective  Technology  Program 


Inputs 


Activities 


Outputs 


Customers  and  intermediate  outcomes 


End 

outcomes 


Production: 

funding  and  staffing; 
physical  infrastructure, 
including  laboratories, 
equipment,3  test  fields,  and 
mobile  units;  a  managerial 
infrastructure,  including 
planning  and  evaluation 
processes 

Planning: 

surveillance  and  risk 
assessments;  strategic- 
planning  documents  (e.g., 
NORA,  r2p);  authorizing 
egulations  (e.g.,  42  CFR  84); 

legislative  mandates; 
COPPE  workforce  report; 

town  hall  meetings; 
stakeholder  input  (e.g., 
emergency  responders, 
employers,  PPE 
manufacturers) 


Respirators 
undergoing  NIOSH 
evaluation, 
investigation,  and/or 
certification  processes 


Develop  and  establish 
criteria,  guidelines,  and 
policy  to  support 
standards;  conduct 
surveillance  and  hazard 
analysis;  conduct 
laboratory  and 
field  research;  conduct 
gap  analyses;  develop  test 
methods;  develop  new 
technology/prototypes; 
conduct  intervention 
effectiveness  evalua¬ 
tions;  conduct  fit- 
testing;  build  capacity; 
train  and  educate 
professionals 


Guidance,  policies,  and 
recommendations;  NIOSH 
reports  and  guidance  docu¬ 
ments;  peer-reviewed  journal 
articles;  other  publications 
(e.g..  Pocket  Guide  to 
Chemical  Hazards);  proto¬ 
types  and  technology; 
patents;  licenses;  copyrights; 
technical  methods,  processes, 
techniques,  tools,  and 
materials;  workshops; 
meeting  presentations; 
education  and  training 
materials;  trained  pro¬ 
fessionals;  software; 

Web  sites 


Develop,  revise,  and 
interpret  policies  and 
standards  relating  to 
respirator  performance, 
quality,  reliability, 
efficacy,  and  design 
Direct  and  carry  out  NIOSH 
Respirator  Certification 

_ Program _ 

Audit  respirators  and 
manufacturing  process; 

conduct  problem 
investigations;  long-term 
field  evaluations 


Policies  and  standards 
(42  CFR  84) 


Respirator  certification 

Investigation  reports, 
compliance  reports,  user 
notices,  recall/retrofit  letter, 
stop-sale  notice,  certification 
revocation,  respirator- 
selection  logic,  certified 
equipment  list 


Intermediate 

customers: 

standard-development 
organizations;  other 
NIOSH  program  areas; 
other  U.S.  agencies; 
state  and  local  entities; 
international  agencies; 
NGOs;  labor,  trade, 
and  professional 
associations;  PPE 
developers/ 
manufacturers; 
independent  test 
organizations  and 
certification 
laboratories 
(e.g.,  SEI,  UL); 
Responder  Knowl¬ 
edge  Base;  academic 
institutions;  health 
and  safety  officers/ 
professionals;  media 


Standards/ 
regulations; 
pilot  and/or 
market-ready 
technologies; 

reports/ 
publications; 
training  and 
education 
programs; 
media  releases; 
Web  sites 


Final 

customers: 

employers, 
workers,  and 
emergency 
responders  who 
rely  on  PPE 


Respirator 

manufacturers 


Partnerships  with  standards-development  organizations  (e.g.,  ISO,  ANSI,  NFPA,  ASTN);  other  NIOSH 
program  areas;  other  U.S.  agencies  (e.g.,  OSHA,  MSHA,  DoD,  NIST,  DHS);  state  health  and  labor 
departments;  local  agencies;  international  agencies  and  organizations;  academic  institutions;  labor, 
trade,  and  professional  associations;  technology  developers/manufacturers;  and  others 


Changes  in 
workplace 
policies, 
practices,  and 
procedures; 
adoption  of 
technologies; 
changes  in 
knowledge, 
attitudes,  and 
behavior 


Reduction  in 
occupational 
injuries, 
illnesses, 
fatalities  and 
hazardous 
exposures 
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Management  objectives 


Annual  goals 


Intermediate  goals 


Strategic 

goals 


External  factors 


SOURCE:  NIOSH  (2007). 

NOTE:  COPPE  =  Committee  on  Personal  Protective  Equipment.  NFPA  =  National  Fire  Protection  Association.  ASTM  =  ASTM  International  (formerly, 
the  American  Society  for  Testing  and  Materials).  DoD  =  U.S.  Department  of  Defense.  NIST  =  National  Institute  of  Standards  and  Technology. 

SEI  =  Software  Engineering  Institute.  UL  =  Underwriters  Laboratories. 

a  NIOSH  laboratories  and  other  facilities  accessed  through  contracts,  cooperative  agreements,  and  contracts. 


o 

IQ 

n' 


O 

Q_ 

n> 


RAND  MG809-3.8 


UJ 


38  Demonstrating  and  Communicating  Research  Impact 


the  research  activities  feed  into  the  activities  of  the  respirator-certification  program.  For 
example,  the  research  activity  to  “develop  and  establish  criteria,  guidelines,  and  policy 
to  support  standards”  directly  feeds  into  the  respirator-certification  program  activity 
of  “develop,  revise,  and  interpret  policies  and  standards  relating  to  respirator  perfor¬ 
mance,  quality,  reliability,  efficacy,  and  design.”  Likewise,  some  of  the  outputs  of  the 
respirator-certification  program  may  generate  activities  for  the  research  program.  For 
example,  investigation  reports  generated  through  the  respirator-certification  program 
may  drive  intervention-effectiveness  evaluations  by  PPT  researchers. 

Both  paths  converge  at  the  level  of  final  customers — employers,  workers,  and 
emergency  responders  who  rely  on  personal  protective  equipment.  In  addition  to  the 
intermediate  outcomes  from  the  research  path  (i.e.,  standards  and  regulations,  pilot 
or  market-ready  technologies,  reports  and  publications,  training  and  educational  pro¬ 
grams,  media  releases,  and  Web  sites)  and  the  respirator- certification  program  (i.e., 
certified  respirators),  some  of  the  respirator-research  outputs  are  delivered  directly  to 
the  final  customers  (e.g.,  Pocket  Guide  to  Chemical  Hazards  [NIOSH,  undated(d)])  as 
indicated  by  the  arrow  connecting  the  research  outputs  to  the  final  customer.  Another 
unique  element  of  the  PPT  program’s  logic  model  is  the  feedback  into  inputs  from 
respirator  tests  that  fail  the  NIOSH  certification  process.  Respirators  that  have  not 
received  NIOSH  certification  must  go  back  through  the  certification  process  before 
they  can  be  used  in  the  workplace. 

The  partnership  box  in  the  PPT  program  logic  model  is  drawn  to  indicate  that 
effective  partnerships  are  crucial  to  the  success  of  the  PPT  program.  Individuals  or 
groups  that  form  partnerships  with  the  PPT  program  are  actively  involved  in  the  pro¬ 
gram’s  activities  and  outputs  and  sometimes  even  assist  in  the  transfer  of  outputs  to 
intermediate  customers.  Therefore,  the  shaded  partnership  box  is  drawn  to  encompass 
activities,  outputs,  and  transfer  to  intermediate  customers. 

Of  all  of  the  logic  models  we  developed  for  the  various  NIOSH  programs,  the 
PPT  logic  model  is  the  most  complex.  Not  only  does  it  depict  the  dual  paths  for 
the  respirator  research  program  and  the  respirator-certification  program,  but  it  also 
depicts  a  more  complex  interaction  with  its  partners  than  the  Construction  and  HHE 
programs  have.  It  also  makes  use  of  several  arrows  to  show  the  interactions  between 
the  two  paths,  a  feedback  arrow  that  brings  some  of  the  intermediate  outcomes  back 
into  the  inputs,  and  a  direct  link  from  PPT  program  outputs  to  the  final  customer. 


Concluding  Thoughts 

Creating  program  logic  models  was  the  first  step  in  preparing  NIOSH  research  pro¬ 
grams  for  external  review  and  helped  to  establish  the  research  program  (rather  than 
projects)  as  the  unit  of  analysis.  In  addition,  the  logic  models  were  used  to  orient  the 
research  programs  to  focus  on  outcomes  rather  than  outputs.  In  the  next  chapter,  we 
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discuss  how  the  logic  models  were  used  to  prepare  outcome  worksheets,  a  tool  for  data 
collection  and  for  tracing  how  specific  research  activities  led  to  end  outcomes. 


CHAPTER  FOUR 


The  Outcome  Worksheet 


The  National  Academies’  framework  document,  described  in  Chapter  Two,  provided 
guidance  on  the  types  of  data  the  NIOSH  research  programs  needed  to  provide  in 
order  to  demonstrate  impact.  For  example,  to  assess  the  outputs  of  research  programs, 
the  framework  document  listed  peer-reviewed  publications  by  program  staff;  peer- 
reviewed  publications  by  external  researchers  funded  by  NIOSH;  NIOSH  reports  in 
the  research  programs;  sponsored  conferences  and  workshops;  databases;  recommen¬ 
dations;  tools,  methods,  or  technologies;  and  patents.  However,  findings  from  phase  I 
of  our  study  indicated  that  most  programs  that  underwent  external  review  found  that 
some  analysis  of  the  information  was  needed  in  order  to  make  sense  of  the  data.  Accord¬ 
ing  to  program  managers  in  other  agencies  whose  programs  had  undergone  reviews, 
simply  handing  over  reams  of  documents  that  represent  evidence  was  not  perceived  to 
be  especially  helpful  to  either  the  reviewers  or  the  program  being  reviewed.  Moreover, 
they  felt  that  programs  should  carefully  consider  which  specific  pieces  of  information 
are  needed  to  demonstrate  impact  and  how  to  convey  the  context  that  will  give  mean¬ 
ing  to  the  data. 

The  guidance  we  provided  to  NIOSH  research  programs  for  helping  to  determine 
the  pieces  of  evidence,  as  well  as  the  broader  context,  is  based  on  the  historical  tracing 
method.  This  method  traces  chronologically  a  series  of  interrelated  events  either  going 
forward  from  the  research  of  interest  to  downstream  outcomes  or  working  backward 
from  an  outcome  along  a  path  that  is  expected  to  lead  to  precursor  research  (Ruegg  and 
Jordan,  2007).  In  conjunction  with  the  logic  models  discussed  in  Chapter  Three,  we 
adapted  this  methodology  to  help  NIOSH  research  programs  describe  the  causal  link 
between  their  activities  in  the  laboratories  and  larger  societal  outcomes.  An  important 
tool  that  supported  the  use  of  this  method  was  the  outcome  worksheet.  The  outcome 
worksheet  was  primarily  designed  as  a  practical  tool  to  help  NIOSH  researchers  think 
through  the  causal  linkages  between  specific  outcomes  and  research  activities,  deter¬ 
mine  the  data  needed  to  provide  evidence  of  impact,  and  provide  an  organizational 
structure  for  the  evidence. 

In  this  chapter,  we  briefly  review  the  evolution  of  historical  tracing  and  its  role  in 
evaluating  and  assessing  R&D  impact.  We  also  describe  the  development  of  the  out- 
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come  worksheets  and  how  they  were  used  to  assist  NIOSH  research  programs  in  trac¬ 
ing  specific  linkages  between  their  research  activities  and  end  outcomes. 


The  Evolution  of  Historical  Tracing 

One  of  the  earliest  examples  of  historical  tracing  was  Project  Hindsight.  DoD  con¬ 
ducted  Project  Hindsight  during  the  early  1960s  to  develop  an  understanding  of  the 
costs  and  benefits  of  supporting  basic  versus  applied  research  (Sherwin  and  Isenson, 
1967).  Project  Hindsight  focused  on  20  major  weapon  systems  supported  by  DoD  and 
traced  the  development  of  each  backward  20  years  to  identify  the  key  research  outputs 
that  contributed  to  each  system’s  realization.  The  findings  of  this  study  suggested  that 
the  returns  to  DoD  for  its  investment  in  basic  research  over  a  15-  to  20-year  time  frame 
were  small  in  comparison  with  other  categories  of  research  (Kreilkamp,  1971).  In  con¬ 
sidering  the  payoff  of  basic  research  over  a  longer  time  frame,  the  study  indicated  that 
basic  research  had  created  value  and  noted  the  contributions  of  atomic  and  nuclear 
physics  as  an  example  of  this.  Officials  of  the  U.S.  Office  of  Science  and  Technology 
Policy  (OSTP)  and  the  NSF  objected  strongly  to  Project  Hindsight’s  methodology,  the 
use  of  research  events  to  determine  the  cost-effectiveness  of  basic  research,  and,  in  par¬ 
ticular,  to  its  short  time  frame  (20  years). 

Project  Hindsight  was  followed  by  the  Technology  in  Retrospect  and  Criti¬ 
cal  Events  in  Science  (TRACES)  study  and  was  sponsored  by  the  NSF  in  1967  (IIT 
Research  Institute,  1968).  In  contrast  to  Project  Hindsight,  the  goal  of  this  study  was 
to  provide  more-detailed  information  on  the  role  of  various  mechanisms,  institu¬ 
tions,  and  types  of  R&D  activity  required  for  successful  technological  innovation.  The 
TRACES  study  used  essentially  the  same  methodology  as  Hindsight,  but  with  a  longer 
time  frame.  The  results  indicated  that  basic  research  had  made  a  substantial  contribu¬ 
tion  to  innovation,  thus  highlighting  the  importance  of  the  time  frame  selected  for 
these  sorts  of  studies. 

In  a  more  recent  version  of  the  TRACES  study,  the  National  Cancer  Institute 
initiated  an  assessment  to  determine  whether  there  were  certain  research  settings  or 
support  mechanisms  that  were  more  effective  in  bringing  about  important  advances  in 
cancer  research  (Narin,  1989).  The  approach  taken  was  analogous  in  concept  to  the  ini¬ 
tial  TRACES  study,  with  the  addition  of  citation  analyses  to  provide  an  independent 
measure  of  the  impact  of  the  papers  associated  with  key  events. 

The  historical-tracing  approach  has  been  used  in  a  widely  adopted  format  known 
as  a  standard  accomplishments  book  (Kostoff,  1994).  Standard  accomplishments  books 
describe  scientific  accomplishments  in  sufficient  detail  for  the  reader  to  understand 
the  science  that  was  accomplished  and  have  some  idea  of  the  potential  importance 
of  the  research  to  the  organizational  mission,  technology,  and,  perhaps,  the  com¬ 
mercial  sectors.  Accomplishments  books  are  not  intended  to  be  all-inclusive,  nor  do 
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they  include  quantitative  estimates  of  impact.  The  accomplishments  are  drawn  from 
the  different  disciplines  funded  by  the  organization  and  are  meant  to  be  portrayed 
as  representative  of  the  breadth  of  activity  (Kostoff,  1997).  The  Office  of  Health  and 
Environmental  Research  (OHER)  at  the  DOE  published  an  accomplishments  book 
that  described  the  40-year  history  of  OHER  and  presented  selected  accomplishments 
in  different  research  areas  that  allowed  the  impacts  and  benefits  of  the  research  to  be 
tracked  through  time  (Kostoff,  1997). 

As  the  methodology  has  evolved,  there  has  been  less  emphasis  on  using  it  to 
determine  the  cost-effectiveness  of  research  and  more  emphasis  on  using  it  to  describe 
accomplishments  and  illustrate  how  investments,  research  activities,  and  outputs  con¬ 
tribute  to  outcomes  that  benefit  the  public.  For  example,  SRI  International  conducted 
case  studies  for  the  NSF  about  the  evolution  of  six  major  engineering  innovations: 
the  Internet,  magnetic  resonance  imaging  (MRI),  reaction  injection  molding  (RIM), 
computer-aided  design  (CAD)  applied  to  electron  circuits,  and  optical  fiber  for  tele¬ 
communication  and  analog  cellular  phones  (Roessner,  1998;  Roessner  et  ah,  1997). 
The  studies  documented  the  consequences  of  the  NSF’s  support  of  research,  educa¬ 
tion,  infrastructure,  supporting  technology,  organizational  leadership,  and  facilitation 
of  technical  communication  and  interaction  for  the  conception,  emergence,  and  com¬ 
mercial  success  of  each  innovation.  The  case  studies  provided  detail  about  the  processes 
by  which  various  modes  of  research  support  contributed  to  the  realization,  over  time, 
of  a  variety  of  valued  outcomes  and  impacts. 

Research-value  mapping  (RVM)  is  another  method  for  tracing  outcomes  to 
research  activities  (Bozeman  and  Roessner,  1995).  However,  RVM  represents  a  signifi¬ 
cant  departure  from  earlier  attempts  to  tell  a  story  about  the  chronology  and  events 
contained  within  the  boundaries  of  a  research  project.  The  RVM  approach  begins  with 
carefully  specified  and  testable  models  of  causation,  as  well  as  a  scheme  for  linking  the 
individual  cases.  This  yields  particularistic  data,  such  as  those  derived  from  a  particu¬ 
lar  case,  and  generalizable  data,  which  come  from  the  quantification  of  elements  across 
cases  (Bozeman  and  Kingsley,  1997).  Despite  its  value  as  one  of  the  few  approaches 
to  historical  tracing  that  uses  both  qualitative  and  quantitative  data,  it  is  inherently 
resource-intensive,  requiring  a  sufficient  number  of  cases  to  permit  quantitative  analy¬ 
sis  and  application  of  inferential  statistics. 

For  the  most  part,  historical  tracing  relies  on  qualitative  data,  including  inter¬ 
views  with  those  involved  in  research  activities,  records  and  descriptions  of  research 
activities,  and  input  from  stakeholders  and  users  of  the  products  generated  from  the 
research.  It  identifies  critical  antecedent  development,  such  as  the  development  of  a 
particular  technology,  or  publishing  of  an  influential  report  that  was  instrumental  to 
achieving  the  outcome.  Key  events,  people,  documents,  and  organizations  are  noted 
and  the  linkages  among  them  established  to  build  a  causal  chain  that,  in  the  end,  dem¬ 
onstrates  causal  contribution.  Critical  questions  that  can  be  addressed  by  this  method 
include  the  following: 
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•  How  have  the  ideas  that  emerged  from  this  research  influenced  subsequent  think¬ 
ing  and  discovery? 

•  What  is  or  has  been  the  path  from  innovation  to  output  to  outcome  and 
impact? 

•  What  were  the  key  factors  influencing  this  path? 

In  drawing  on  the  historical  tracing  method,  we  augmented  it  with  the  pro¬ 
gram  logic  model  to  increase  the  transparency  of  the  plausible  causal  linkages  between 
research  activities  and  outcomes.  Thus,  the  significance  of  key  outputs  is  based  on  their 
contribution  to  outcomes,  which  is  supported  by  documentation  from  customers  and 
stakeholders.  Tracing  the  specific  impact  pathway  between  research  activities  and  out¬ 
comes  also  relied  on  identifying  and  gathering  the  appropriate  data  to  substantiate  the 
causal  linkages  in  the  path  to  end  outcomes.  To  achieve  this,  we  developed  outcome 
worksheets  based  on  the  logic  model. 


Developing  Outcome  Worksheets:  Building  from  Logic  Models 

The  first  step  in  developing  the  outcome  worksheet  is  deciding  whether  to  trace  the 
research  path  forward  (i.e.,  from  research  activities  to  outcomes)  or  backward.  Forward 
tracing  can  capture  a  complete  view  of  a  research  project’s  or  program’s  effects,  and, 
because  the  path  leads  from  the  research,  the  connection  to  the  research  is  ensured.  An 
example  of  the  use  of  forward  tracing  is  a  recent  study  that  used  the  Buxton-Hanney 
payback  model  to  trace  the  returns  from  arthritis  research  funded  by  the  Arthritis 
Research  Campaign  to  its  outcomes  (Wooding  et  ah,  2004).  Case  studies  of  16  research 
grants  were  conducted  and  written  up  as  a  narrative  organized  according  to  the  struc¬ 
ture  provided  by  the  payback  framework.  Cross-case  analysis  was  conducted  to  assess 
the  extent  to  which  different  types  of  funding  support  might  prevent  or  promote  the 
successful  translation  of  research.  As  such,  forward-tracing  studies  may  be  of  interest 
to  parties  investing  in  or  undertaking  research  (Wooding  et  ah,  2005). 

Backward  tracing  usually  focuses  on  a  single  outcome  of  importance  and  follows 
the  trail  back  through  those  developments  that  were  critical  in  reaching  the  identified 
outcome.  One  implication  of  backward  tracing  is  that  it  highlights  activities  that  led  to 
anticipated  outcomes  and  may  not  capture  the  broader  range  of  outcomes  that  forward 
tracing  may  lead  to  or  may  select  only  the  most  positive  cases  of  outcomes.  However, 
the  National  Academies’  reviews  focused  on  impact,  so  outcomes  were  the  natural 
starting  point,  followed  by  customers,  transfer  activities,  outputs,  research  activities, 
and,  finally,  inputs.  Not  only  did  this  backward  tracing  reinforce  the  emphasis  on 
outcomes,  it  also  oriented  researchers  to  focus  on  a  collective  body  of  research  rather 
than  on  individual  research  projects.  Outcomes  typically  flow  from  a  collective  body 
of  research  activities  that  may  occur  over  several  years,  not  an  individual  research  proj- 
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ect.  With  rare  exceptions,  individual  research  projects  will  most  likely  lead  to  outputs 
(such  as  journal  articles,  recommendations,  and  findings)  but  seldom  result  in  inter¬ 
mediate  outcomes  (such  as  changes  in  practice  by  an  employer  or  employee).  However, 
collectively,  a  body  of  research  conducted  during  a  specific  time  frame  is  likely  to  lead 
to  significant  outcomes. 

To  prompt  program  staff  to  begin  to  think  backward  from  outcomes  to  earlier 
stages,  we  created  a  graphic  to  illustrate  this  reversal.  Figure  4.1  illustrates  how  the 
logic  model  path,  beginning  with  inputs,  was  reversed  (i.e.,  beginning  with  end  out¬ 
comes)  to  create  the  outcome  worksheet. 

The  logic  model  begins  with  the  “Inputs”  box  at  the  left  end  and  ends  with  the 
“End  outcomes”  box  in  purple.  The  outcome  worksheet  (shown  in  the  lower  half  of 
Figure  4.1)  reverses  the  order  of  the  logic  model  elements,  with  the  end  outcomes  at  the 
left  end  and  the  inputs  at  the  right.  This  reversal  of  order  moves  beyond  the  program 
theory  articulated  in  the  logic  model  and  essentially  places  the  burden  on  research  pro¬ 
grams  to  trace  backward  how  specific  outcomes  were  generated  from  research  activi¬ 
ties.  The  colored  bars  within  each  of  the  columns  represent  the  text  that  describes  a 
particular  logic  model  element  (e.g.,  inputs,  activities).  Because  our  purpose  in  creating 
the  outcome  worksheet  was  to  help  the  research  programs  think  through  the  impact 

Figure  4.1 
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pathway — or  path  to  outcomes — in  as  much  detail  as  possible,  each  column  corre¬ 
sponds  to  an  element  of  the  logic  model,  including  the  transfer  arrow.  For  an  impact 
evaluation,  it  is  important  to  explicate  the  role  of  intermediaries  in  the  uptake,  adop¬ 
tion,  and  use  of  research  outputs  (Cooksey,  2006).  Thus,  we  distinguished  between 
the  two  categories  of  customers  (intermediate  and  final)  as  well  as  the  corresponding 
categories  of  outcomes  (intermediate  and  end).  The  “Intermediate  outcomes”  column 
with  the  purple  bars  corresponds  to  the  response  and  actions  taken  by  final  customers, 
and  the  column  with  light-blue  bars  corresponds  to  the  response  and  actions  taken  by 
the  intermediate  customers.  Despite  the  redundancy  in  terminology,  this  detailed  pars¬ 
ing  of  information  ensured  transparency  of  the  linkages  between  information  across 
the  columns.  The  inclusion  of  strategic  and  intermediate  goals  in  the  outcome  work¬ 
sheet  is  based  on  guidance  from  the  framework  document,  which  recommended  that 
the  ECs  evaluate  the  research  program  goals  and  objectives  and  provide  a  qualitative 
assessment  of  the  relevance  of  the  program’s  goals,  objectives,  and  strategies  in  relation 
to  its  major  challenges.  Because  our  logic  model  template  (Figure  3.3  in  Chapter  Three) 
links  and  aligns  program  operations  with  program  strategy,  inclusion  of  the  goals  in 
the  outcome  worksheet  reinforced  the  supportive  relationship  between  program  opera¬ 
tions  (i.e.,  inputs,  activities,  outputs,  and  outcomes)  and  program  goals. 

An  example  of  the  type  of  information  included  in  the  outcome  worksheet  can 
be  found  in  the  HHE  program.  In  this  program,  facilities  that  are  being  investigated 
by  the  HFIE  research  program  are  one  category  of  final  customers,  and  the  names  of 
these  facilities  would  be  listed  in  the  column  labeled  “Final  customers.”  Similarly,  in 
the  “Intermediate  outcomes”  column  to  the  left,  specific  descriptions  of  adoption  of 
technologies;  changes  in  workplace  policies,  practices,  and  procedures;  or  changes  in 
the  physical  environment  and  organization  of  work  linked  to  these  final  customers 
would  be  appropriate  information  to  include  in  this  column.  An  example  of  an  inter¬ 
mediate  outcome  from  the  Hearing  Loss  Prevention  program  is  the  “manufacture  of 
plastic  coated  flight  bars  for  continuous  mining  machines”  by  Joy  Mining  Machinery, 
a  continuous-miner  manufacturer.  The  manufacture  of  the  coated  flight  bars  is  based 
on  research  activities  in  the  Hearing  Loss  Prevention  program  to  develop  and,  ulti¬ 
mately,  produce  a  chain  conveyor  with  coated  flights  as  a  noise  control. 

Note  that,  in  Figure  4.1,  there  is  one  bar  underneath  the  “End  outcomes”  column 
and  multiple  bars  in  the  other  columns,  with  the  greatest  number  of  bars  in  the  “Inputs” 
column.  Pictorially,  this  reinforces  an  earlier  point — namely,  that  an  end  outcome  will 
often  derive  from  a  body  of  research  rather  than  from  a  single  research  activity.  For 
example,  an  end  outcome  of  a  10-percent  reduction  in  construction  workers’  asbestos 
exposure  will  likely  result  from  multiple  research  activities,  such  as  field  studies  to  assess 
the  prevalence  of  asbestos  exposure,  intervention  studies  to  document  the  most  cost- 
effective  ways  to  minimize  exposure,  and  perhaps  population  studies  to  document  con¬ 
struction  workers  who  are  at  higher  risk  for  asbestos  exposure.  Each  of  these  research 
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activities  are  likely  to  lead  to  several  outputs  that  are  transferred  to  different  customers, 
leading  to  intermediate  outcomes  that  serve  as  precursors  to  the  end  outcome. 


Gathering  Data  for  the  Outcome  Worksheet 

We  asked  program  managers  to  identify  the  most-significant  outcomes  that  occurred 
during  the  period  under  review.  The  program  managers  shared  information  with  the 
program  researchers.  The  number  of  outcomes  varied  across  programs.  In  the  Hear¬ 
ing  Loss  Prevention  program,  44  outcomes  were  initially  identified  across  10  major 
research  areas,  whereas  in  the  Traumatic  Injury  research  program,  approximately  23 
outcomes  were  identified.  To  conform  to  the  guidance  from  the  framework  document, 
we  asked  the  research  programs  to  group  the  outcomes  by  strategic  goals.  As  described 
in  Chapter  Three,  our  logic  model  template  shows  that  end  outcomes  and  strategic 
goals  should  be  mutually  supportive.  For  example,  one  of  the  strategic  goals  of  the 
Traumatic  Injury  program  is  to  reduce  occupational  injuries  and  fatalities  in  Alaska. 
End  outcomes  that  support  the  achievement  of  that  goal  include  reducing  occupational 
injuries  and  fatalities  in  commercial  fishing,  helicopter  logging  operations,  and  Alaska 
aviation.  This  connection  between  the  program’s  strategic  goals  and  their  outcomes 
was  an  important  structural  feature  that  created  coherence  within  the  research  pro¬ 
gram  description. 

Identifying  outcomes  was  the  starting  point  for  gathering  evidence  to  demonstrate 
impacts.  After  listing  the  intermediate  or  end  outcomes,  we  asked  programs  to  fill  in 
the  box  immediately  to  the  right.  In  the  case  of  end  outcomes,  the  next  box  contains 
the  intermediate  outcomes.  In  cases  in  which  programs  could  report  only  intermediate 
outcomes,  the  next  box  contains  customers  (final  or  intermediate).  For  the  most  part, 
the  research  programs  did  not  complete  the  outcome  worksheet  in  a  linear  fashion  but 
rather  filled  in  the  cells  with  the  most-accessible  information  first.  Not  surprisingly, 
linking  research  activities  to  outputs  and  identifying  the  research  activities  and  out¬ 
puts  that  could  be  associated  with  the  significant  outcomes  were  fairly  straightforward. 
However,  determining  the  path  of  how  the  outputs  contributed  or  led  to  an  outcome 
required  more  time  and  effort. 

Each  program  employed  its  own  strategy  for  gathering  data  to  complete  the  out¬ 
come  worksheets.  These  approaches  included  reviewing  their  own  transfer  activities  in 
reaching  intermediate  and  final  customers,  directly  contacting  intermediate  customers 
(i.e.,  industry,  trade  associations,  other  federal  agencies,  state  and  local  OSH  agencies) 
to  find  out  how  specific  outputs  had  been  used;  contacting  partners  to  find  out  whether 
they  were  aware  of  and  could  cite  changes  in  the  workplace  based  on  program  work; 
following  up  with  individuals  or  organizations  that  requested  information  from  the 
research  program;  using  search-engine  tools  (i.e.,  Google®  search,  LexisNexis®,  Public 
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Library  of  Science)  to  identify  hits  from  searches  for  NIOSH  report  titles;  and  identify¬ 
ing  patented  technologies  related  to  the  specific  research  program. 


Analyzing  Information  in  the  Outcome  Worksheet 

In  addition  to  identifying  and  structuring  information,  the  format  of  the  outcome 
worksheets  enables  quick  review  and  analysis  of  a  large  amount  of  information.  Specifi¬ 
cally,  we  reviewed  the  outcome  worksheets  to  determine  whether  (1)  information  was 
missing  or  inappropriately  categorized  and  (2)  information  across  the  columns  could 
be  causally  linked.  Figure  4.2  provides  a  schematic  of  an  initial  draft  of  the  outcome 
worksheet. 

In  Figure  4.2,  some  cells  are  completely  empty,  which  is  an  indication  that  some 
information  in  the  causal  path  is  missing.  However,  the  outcome  worksheet  does  more 
than  indicate  that  information  is  missing.  It  also  identifies  specifically  which  type  of 
information  is  missing  in  the  causal  path.  Is  it  the  intermediate  outcome  from  the 
final  customer?  If  so,  this  can  cue  program  managers  about  whom  to  contact  in  order 
to  track  down  this  information.  Other  cells  contain  a  mix  of  different-colored  bars, 
which  indicates  inappropriately  categorized  information.  Similar  to  developing  the 
logic  models,  determining  the  appropriate  category  for  some  information  is  not  always 

Figure  4.2 

A  Schematic  of  an  Initial  Draft  of  an  Outcome  Worksheet 
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easy.  For  example,  a  presentation  at  a  conference,  should  that  be  considered  an  output 
or  a  transfer  activity?  It  depends,  in  part,  on  the  audience  and  purpose  of  the  presenta¬ 
tion.  Discerning  incorrectly  categorized  information  is  easier  to  accomplish  when  the 
information  is  in  the  form  of  a  worksheet  on  which  the  causal  thread  can  be  followed. 
Thus,  a  major  advantage  of  the  outcome  worksheet  is  that  it  helped  research  programs 
address  both  the  questions,  “Do  I  have  the  right  information?”  and  “Is  the  information 
in  the  right  place?” 

Displaying  the  linkages  between  information  in  the  columns  is  one  of  the  most 
important  functions  of  the  outcome  worksheet.  Figure  4.3  shows  an  early  draft  of  an 
outcome  worksheet  that  was  prepared  from  the  Hearing  Loss  Prevention  program. 

Note  the  causal  linkages  between  the  activity  of  “Construct  a  hearing-protector 
testing  lab,”  the  output  of  “Testing  laboratory  for  hearing  protector,”  and  the  transfer 
activity  of  “Access  to  laboratory,”  which  is  highlighted  in  orange  in  Figure  4.3.  Beyond 
this  point,  the  link  from  this  activity  to  the  end  outcome  of  decreased  noise  exposure 
to  EPA  workers  is  unclear.  In  cases  like  this,  the  program  managers  most  likely  con¬ 
ducted  follow-up  research  to  determine  whether  there  was  a  connection  to  the  end  out¬ 
come.  For  example,  the  program  manager  may  contact  the  intermediate  customer — 
U.S.  Army  Aero  medical  Research  Laboratory — to  determine  whether  it  conducted  any 
testing  at  the  hearing-protector  laboratory  and,  if  so,  what  the  results  were  and  whether 
they  led  to  specific  outcomes.  Generally,  if  the  programs  could  gather  data  indicating 
a  response  from  the  customer,1  the  information  was  included  as  part  of  the  outcome 
worksheet.  If  not,  the  information  may  have  been  removed  from  the  outcome  work¬ 
sheet  and  included  in  a  discussion  of  emerging  research  activities  in  the  evidence  pack¬ 
age.  Similarly,  note  the  linkages  in  blue  between  the  output  of  the  EarTalk  system  and 
the  intermediate  outcome  of  the  marketed  EarTalk  system.  In  this  path,  the  research 
activities  and  resources  that  contributed  to  this  outcome  need  to  be  established. 

At  the  end  of  the  analysis,  our  goal  was  to  have  outcome  worksheets  with  informa¬ 
tion  filled  in  each  cell,  properly  aligned  in  the  correct  columns  and  clearly  established 
causal  links  between  the  information  in  the  columns.  Figure  4.4  presents  a  schematic 
of  what  the  final  version  of  an  outcome  worksheet  should  look  like. 


1  The  Framework  Document  (National  Academies,  2007)  defines  outcomes  as  the  customer’s  response  to 
outputs. 
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Outcome  Worksheet:  Hearing  Loss  Prevention  Program 
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Figure  4.4 

A  Schematic  of  a  Final  Draft  of  an  Outcome  Worksheet 
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Concluding  Thoughts 

Logic  models  and  outcome  worksheets  were  used  as  analytical  tools  in  preparing  for 
external  review.  Logic  models  promoted  thinking  through  and  describing  impact 
pathways.  Outcome  worksheets  helped  trace  specific  instances  of  programs’  contribu¬ 
tions  to  outcomes.  The  next  chapter  focuses  on  the  evidence  package  and  the  tool  used 
to  assist  in  communicating  impact.  There,  the  logic  model  and  outcome  worksheets 
played  important  roles.  The  logic  model  helped  to  determine  the  structure  of  the  evi¬ 
dence  package,  and  the  outcome  worksheets  were  used  to  create  the  key  section  of  the 
evidence  package:  the  outcome  narrative.  The  next  chapter  covers  the  different  ele¬ 
ments  and  creation  of  the  evidence  package  in  detail. 


CHAPTER  FIVE 


The  Evidence  Package 


As  the  final  part  of  our  implementation  phase,  we  helped  the  NIOSH  programs 
develop  a  package  of  information  to  give  to  reviewers:  the  evidence  package.  Each 
evidence  package  included  the  data  requested  by  the  National  Academies  in  the  frame¬ 
work  document  as  well  as  additional  materials  that  the  program  deemed  necessary  to 
demonstrate  impact.  The  outcome  narrative  is  at  the  heart  of  the  package.  It  provides 
a  detailed  account  of  how  research  inputs,  activities,  and  outputs  led  to  specific  out¬ 
comes.  In  this  chapter,  we  discuss  the  evidence  package  and  present  guidance  for  com¬ 
municating  the  impacts  of  scientific  research  in  the  outcome  narratives. 


Communicating  Impact 

The  central  purpose  of  the  evidence  package  is  to  communicate  to  reviewers  that  activi¬ 
ties  have  contributed  to  societal  outcomes.  The  reviewers  were  expected  to  use  their 
expert  judgment  and  knowledge  of  the  field  to  evaluate  the  claims  in  the  evidence 
package  about  the  role  of  NIOSH  programs  in  achieving  intermediate  outcomes  (such 
as  changes  in  workplace  practices)  or  end  outcomes  (such  as  reductions  in  hazardous 
exposures).  Thus,  the  information  in  the  evidence  package  is  designed  to  convey  and 
support  claims  of  impact.  In  this  way,  an  evidence  package  differs  from  the  drafting  of 
the  more-familiar  research  products,  such  as  journal  articles,  research  reports,  or  other 
products.  In  the  following  sections,  we  describe  these  differences. 

Communicating  to  a  Different  Audience 

Two  key  elements  in  communication  are  the  audience  and  the  communicator’s  pur¬ 
pose  in  communicating  with  that  audience  (Kinneavy,  1971).  The  starting  point  in 
creating  the  evidence  package  is  thinking  about  the  intended  audience.  In  this  case, 
the  audience  is  the  expert  reviewers.  Even  though  the  reviewers  may  well  be  researchers 
themselves,  they  will  be  reading  the  evidence  package  from  a  different  perspective — 
as  evaluators.  As  members  of  a  reading  audience,  researchers  and  evaluators  are  likely 
to  differ  in  fundamental  ways.  To  explore  these  differences  and  their  implications,  we 
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developed  a  generic  portrait  of  how  researchers  and  evaluators  are  likely  to  differ  as 
audiences  (see  Table  5.1).1 

First,  the  level  of  knowledge  about  the  subject  at  hand  may  differ  significantly 
across  these  audiences.  Researchers  reading  journal  articles  or  reports  in  their  own 
fields  of  research  are  likely  to  have  substantial  expertise  in  the  field  and  to  know  just  as 
much  as  the  research  author  about  the  field  itself.  By  contrast,  evaluators’  knowledge 
of  the  field  may  vary  considerably.  The  evaluator  could  well  be  a  researcher  who  under¬ 
stands  a  great  deal  about  the  field,  but  it  is  more  likely  that  the  evaluator  is  less  knowl¬ 
edgeable  about  the  research  field  per  se;  in  fact,  the  evaluator  may  not  be  a  researcher  at 
all,  but  rather  a  policymaker,  industry  stakeholder,  labor-union  representative,  or  some 
other  kind  of  stakeholder.  These  readers  are  less  steeped  in  the  research  methods  and 
discourse  but  likely  to  be  highly  knowledgeable  about  the  real-world  contexts  in  which 
the  research  is  intended  to  make  a  difference. 

Evaluators  also  use  information  differently.  Researchers  poring  over  journal  arti¬ 
cles  or  reports  are  typically  trying  to  improve  their  understanding  of  the  field,  keep 
abreast  of  the  latest  findings,  and  understand  methodological  challenges  involved  in 
conducting  studies.  In  contrast,  evaluators  read  for  a  much  narrower,  more  focused 
reason:  to  assess  the  validity  of  claims  about  the  outcome  of  research,  whether  the 


Table  5.1 

A  Tale  of  Two  Audiences:  Researchers  and  Evaluators 


Characteristic 

Researcher 

Evaluator 

Knowledge  of  subject 

Usually  vast 

Variable,  ranging  from  vast  to 
sketchy 

How  they  use  information 

To  increase  understanding 

To  make  a  judgment  or  decision 

Attitude  toward  subject 

Often  desire  to  understand  topic 
in  detail 

Want  to  know  how  information  is 
relevant  to  them 

Time  constraints 

Take  maximum  time  possible  to 
study  topic 

Want  to  digest  information  as 
quickly  as  possible 

Focus 

Validity  of  findings 

Causal  links:  how  studying 
the  problem  leads  to  societal 
improvement 

Purpose  of  communicating  to 
audience 

Convey  findings  of  research 

Demonstrate  impact  of  research 

Nature  of  problem 

Research 

Societal 

Desired  result 

Advance  knowledge  in  research 
field 

Inform  judgment  about  research 
impact 

1  Much  of  the  information  in  this  table  derives  from  conversations  with  communication  analysts  at  RAND, 
whose  distilled  wisdom  can  be  found  in  Guidelines  for  Preparing  Briefings  (1996). 
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claims  made  in  the  evidence  package  are  valid  and  supported  by  the  evidence  pre¬ 
sented,  and,  ultimately,  the  impact  of  the  research. 

These  audiences  also  have  different  attitudes  about  the  subject  and  a  different 
time  frame  in  which  the  information  is  used.  Researchers  possess  an  intrinsic  inter¬ 
est  in  research  methods  and  results  and  want  to  understand  them  in  detail.  They  are 
likely  to  immerse  themselves  in  presentations  of  findings.  However,  evaluators  tend 
to  approach  information  with  a  decisionmaking  orientation.  They  have  a  judgment  to 
make  and  approach  the  material  with  an  eye  on  its  relevance  to  the  assessment.  Evalu¬ 
ators  want  to  absorb  the  information  more  quickly  than  they  might  if  reading  from  a 
research  orientation.  They  have  been  convened  for  a  specific  task  and  want  to  complete 
it  as  expeditiously  as  possible. 

Along  with  these  differences  comes  a  difference  in  focus.  Researchers  reading  a 
research  publication,  especially  as  peer  reviewers,  focus  primarily  on  methodological 
validity  and  the  soundness  of  the  hndings.  Evaluators  of  evidence  packages  that  focus 
on  impact  are  focused  on  the  causal  links  between  the  study  of  a  particular  problem 
and  outcomes  that  represent  societal  improvement. 

Communicating  for  a  Different  Purpose 

It  follows  from  these  differences  in  audience  orientation  that  the  author  of  an  evidence 
package  has  a  more  practical  purpose  in  mind:  informing  the  evaluators’  decision  about 
whether  the  package  has  made  a  convincing  case  for  the  impacts  it  has  traced  back  to 
the  research  program  under  evaluation.  Thus,  evidence-package  authors  must  answer 
the  following  questions  for  their  audience: 

•  So  what ?  How  is  this  information  going  to  help  me  assess  impact? 

•  Who  cares ?  Whom  does  this  affect? 

•  Who  says ?  What  evidence  do  you  have  to  substantiate  your  claim  that  links  your 

research  to  outcomes? 

•  And  your  point  is?  What  outcomes  have  you  achieved? 

Readers  of  evidence  packages  must  come  to  a  decision  about  whether  the  evi¬ 
dence  presented  support  claims  of  impact.  This  decisionmaking  orientation  means  that 
evaluators  are  looking  for  clear,  concise,  and  accessible  evidence  packages  that  can  be 
easily  absorbed.  Thus,  simply  collecting  information  (such  as  project  descriptions)  or 
outputs  (such  as  journal  articles)  into  an  evidence  package  in  the  hopes  that  the  “reader 
will  make  the  necessary  connections  and  do  the  hard  work  of  tying  the  information 
together  should  be  avoided”  (Montgomery,  2003). 

In  the  next  section,  we  describe  the  guidance  we  shared  with  the  NIOSH  research 
programs  for  constructing  the  evidence  package,  leading  to  a  more  narrowly  focused 
discussion  of  the  key  section,  the  outcome  narrative. 
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The  Structure  of  the  Evidence  Package 

The  evidence  packages  that  the  NIOSH  research  programs  prepared  followed  a  stan¬ 
dard,  uniform  structure  that  mirrored  the  logic  model.2  As  the  conceptual  blueprint 
for  how  the  research  program  gets  to  outcomes,  the  logic  model  structure  provided  the 
basic  organizational  sequence  of  the  evidence  package,  concluding  with  an  account  of 
outcomes  and  how  the  research  contributed  to  them. 

Most  of  the  evidence  packages  consisted  of  the  following  sections:  (1)  introduction 
to  NIOSH,  (2)  overview  of  the  research  program,  (3)  research  program  strategic  goals 
and  outcomes,  and  (4)  vision  for  the  future.  The  Respiratory  Diseases  Research  Pro¬ 
gram  (RDRP)  described  the  outline  of  its  evidence  package  in  the  following  manner: 

The  first  two  chapters  of  the  evidence  package  provide  introductions  to  NIOSH 
and  RDRP.  The  next  four  chapters  are  disease-focused  and  address  interstitial  lung 
diseases,  airways  diseases,  respiratory  malignancies,  and  infectious  diseases.  An 
exposure-focused  chapter  addresses  nanotechnology  research.  Finally,  five  activity- 
focused  chapters  address  some  of  the  more  prominent  institute  activities  needed 
to  reduce  work-related  respiratory  diseases.  These  activities  include  surveillance, 
[HETAB],  emergency  response  and  disaster  preparedness,  respirator  policy,  and 
sampling  and  analytical  methods  activities.  A  final  chapter  presents  the  RDRP 
vision  for  the  future.  (NIOSH  undated[e]) 

The  first  section  described  NIOSH,  including  its  organizational  structure,  mis¬ 
sion,  value,  and  strategic  plans.  It  also  included  fiscal,  staffing,  and  planning  resources. 
The  second  section  focused  on  the  specific  research  program  being  reviewed — in  this 
case,  RDRP — and  provided  much  of  the  information  that  the  framework  document 
outlined  for  describing  inputs,  activities,  and  outputs.  This  section  discussed  the  pro¬ 
duction  resources,  such  as  personnel,  funding,  facilities,  and  equipment  used  by  the 
research  program.  It  also  included  planning  resources,  such  as  prior  program  evalu¬ 
ations  of  the  research  program,  strategic-planning  processes,  and  stakeholder  input. 
The  program’s  logic  model  was  included  in  the  second  section  along  with  a  description 
of  the  types  of  activities,  outputs,  transfer,  and  customers  of  the  program.  The  third, 
and  usually  the  largest,  section  was  the  one  on  outcomes.  In  the  case  of  RDRP,  this 
section  of  the  evidence  package  consisted  of  four  chapters  organized  by  disease  focus. 
It  featured  the  outcome  narratives,  which  presented  the  evidence  for  impact  and  rel¬ 
evance  by  detailing  how  specific  areas  of  research  activities  led  to  intermediate  or  end 
outcomes.  The  narrative  section  is  the  centerpiece  of  the  evidence  package.  In  effect,  it 
provides  the  links  across  the  components  of  the  logic  model.  As  such,  it  must  do  much 
of  the  package’s  heavy  lifting:  It  must  demonstrate  the  case  for  the  research  leading  to 


2  Links  to  the  NIOSH  research  program  evidence  packages  can  be  found  at  NIOSH  (2008c). 
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outcomes  that  will  provide  the  evaluators  with  the  evidence  of  impact  they  have  been 
asked  to  judge. 


Using  Outcome  Worksheets  to  Prepare  Outcome  Narratives 

The  outcome  narrative  consists  of  a  standard  template  that  includes  the  following  five 
sections:  Issue,  Approach,  Outputs  and  Transfer,  Intermediate  (and  End)  Outcomes, 
and  What’s  Ahead.  Each  section  includes  a  specific  set  of  information.  For  example, 
the  Issue  section  describes  the  major  problem  or  considers  what  the  research  was  trying 
to  address,  whereas  Outputs  and  Transfer  focuses  specifically  on  the  research  products 
and  how  they  are  transferred  to  customers. 

Figure  5.1  shows  how  the  information  from  the  outcome  worksheet  was  used  to 
create  the  outcome  narrative. 

As  described  earlier,  the  colored  bars  indicate  specific  types  of  information  that 
correlate  with  the  logic  model.  Thus,  the  Issue  section  draws  largely  from  the  informa¬ 
tion  on  inputs,  as  they  provide  both  the  resources  for  conducting  research  activities  and 
the  guidance  for  determining  which  research  activities  to  conduct.  The  Approach  sec¬ 
tion  describes  the  research  activities  that  were  conducted  to  address  the  issue.  The  Out¬ 
puts  and  Transfer  section  includes  four  colors,  corresponding  to  outputs,  transfer,  and 
intermediate  and  final  customers.  Each  of  these  logic  model  elements  are  included  in 
the  Outputs  and  Transfer  section,  as  it  describes  specific  outputs,  the  transfer  activities, 
and  the  customers  who  were  the  recipients  of  the  transfer  activities.  Finally,  the  Inter¬ 
mediate  (and  End)  Outcomes  section  describes  the  specific  customers  who  received  the 
outputs  and  the  response  to  the  outputs. 

Writing  the  Outcome  Narrative 

The  outcome  worksheet  provides  a  skeletal  structure  for  the  outcome  narrative.  Here, 
we  fill  in  details  about  creating  the  sections  of  the  narrative  and  provide  some  examples 
of  effective  narrative  sections. 

Issue:  What  is  the  major  societal  problem?  The  narrative  should  begin  by  clearly 
and  concisely  defining  the  issue  and  its  significance,  why  it  exists,  and  who  is  affected 
(Poling,  undated).  In  the  process  of  defining  the  issue,  baseline  data  are  helpful,  as  they 
help  to  define  the  scope  of  the  problem.  The  focus  should  be  on  the  issue  area  per  se 
rather  than  the  events  that  led  to  awareness  of  the  issue.  For  example,  in  describing  the 
problem  of  workplace  hearing  loss,  it  may  be  helpful  but  is  not  necessary  to  describe 
how  understanding  of  the  problem  came  to  light  or  evolved;  it  is  necessary  only  to 
describe  the  problem  itself.  It  is  also  helpful  to  provide  context  that  explains  why  this 
issue  is  important  (such  as  the  magnitude  of  the  problem,  e.g.,  the  number  of  workers 
at  risk  and  the  severity  of  the  risk)  and  to  describe  why  the  issue  exists  (for  example, 
inadequate  standards,  lack  of  technology,  tools,  or  materials  to  minimize  risk,  lack  of 
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Figure  5.1 

Translating  the  Outcome  Worksheet  to  the  Outcome  Narrative 


Outcome  worksheet 


Strategic 

goals 

Inter¬ 

mediate 

goals 

End 

outcomes 

Inter¬ 

mediate 

outcomes 

Final 

customers 

Inter¬ 

mediate 

outcomes 

Inter¬ 

mediate 

customers 

Transfer 

Outputs 

Activities 

Inputs 

— 

— 

— 

— 

— 

1111 

Outcome  narrative 


* 


RAND  MG809-5.1 


knowledge,  awareness  or  skills).  A  major  challenge  in  this  area  is  ensuring  that  the  reso¬ 
lution  of  the  issue  is  clearly  linked  to  the  program’s  end  outcome.  It  is  not  sufficient  to 
provide  evidence  that  the  problem  has  been  reduced — there  must  also  be  a  clear  link 
between  this  outcome  and  the  research  program.  Finally,  it  is  important  to  describe  the 
portion  of  the  issue  on  which  the  program  is  focused  and  to  describe  how  the  program’s 
work  in  this  area  will  address  the  issue. 

An  excerpt  from  NIOSH’s  HHE  evidence  package  provides  an  example  of  an 
effective  issue  statement: 


Biological  Hazards.  Until  the  early  1990s,  thousands  of  American  cities  were 
allowed  to  dump  their  raw  and  treated  domestic  sewage  directly  into  the  nation’s 
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rivers,  lakes,  and  bays.  By  1992,  EPA  enacted  regulations  to  stop  this  practice,  and 
municipalities  began  separating  the  domestic  sewage  into  liquid  and  solid  streams. 

The  treated  solid  component,  referred  to  as  biosolids,  was  approved  for  use  as  an 
agricultural  soil  amendment  and  crop  fertilizer,  and  in  reclamation  of  strip  mines. 
Shortly  after  the  EPA  regulations  were  enacted,  questions  were  raised  about  poten¬ 
tial  health  risks  to  workers  involved  in  land  application  of  biosolids  and  to  people 
living  in  surrounding  communities.  In  1994,  the  death  of  a  child  from  staphylo¬ 
coccal  septicemia  one  week  after  riding  a  three-wheeler  over  biosolids  sparked  con¬ 
cern  in  the  local  community,  extensive  media  coverage,  and  attention  by  several 
employee  groups,  including  the  United  Mine  Workers. 

Based  on  data  from  2002,  an  estimated  3.4  million  dry  tons  of  biosolids  were 
used  for  agricultural  purposes  that  year.  Exposed  occupational  populations  include 
wastewater  treatment  workers,  haulers,  applicators,  and  farmers.  In  2004,  the  BLS 
estimated  that  there  were  94,000  water  and  wastewater  treatment  plant  and  system 
operators  in  the  U.S. 

In  February  1998,  the  HHE  Program  received  a  request  from  the  safety  manager 
at  the  Butler  County  Department  of  Environmental  Services  in  Ohio  for  assis¬ 
tance  in  evaluating  workers’  exposures  during  the  processing  and  land  application 
of  biosolids.  The  EIE1E  request  stated  that  some  long-term  employees  had  reported 
headaches,  stomach  cramps,  and  diarrhea  after  working  with  the  biosolids.  The 
HHE  Program  also  received  two  subsequent  HHE  requests  in  1999  and  2002 
to  evaluate  worker  exposures  during  the  handling  of  biosolids  as  part  of  mine 
reclamation  activities  and  spreading  of  biosolids  at  the  working  face  of  a  landfill. 
(NIOSH,  undated[c]) 

This  statement  furnishes  the  important  information  concisely  and  clearly.  The 
problem  (exposure  to  hazardous  biosolids  in  the  workplace)  is  defined,  and  its  signifi¬ 
cance  is  stated  in  terms  of  the  estimated  number  of  workers  exposed. 

Approach:  What  approach  has  been  used  to  address  this  issue?  This  section 
describes  the  research  strategies  that  have  been  used  to  address  the  issue.  Ideally,  it 
should  move  from  general  to  specific.  It  should  present  the  overall  strategy  before 
describing  specific  activities.  The  section  should  also  explain  how  the  methodology 
used  by  the  program  in  question  differs  from  others  (such  as  partners  or  stakehold¬ 
ers)  who  may  also  work  in  this  area.  Use  of  the  active  voice  in  the  approach  section  is 
important  so  that  it  is  clear  who  did  what.  Although  specificity  in  discussing  approach 
and  methodology  is  necessary,  project  descriptions  and  excessive  research  details  should 
be  avoided. 

This  section  should  not  present  merely  the  program’s  broad  strategies  or  general 
action  plan.  The  broad  strategies  may  tackle  only  a  subset  of  the  motivating  conditions 
and  causes.  The  strategies  really  represent  a  cluster  of  specific  activities,  interventions, 
or  services  that  the  program  will  undertake  to  implement  the  overall  strategies.  These 
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different  strands  need  to  be  articulated  clearly  and  separately.  Again,  we  furnish  an 
example  from  the  HHE  research  program’s  evidence  package: 

Approach.  To  determine  the  source  of  the  illnesses,  HHE  investigators  conducted 
monitoring  to  assess  whether  workers  were  exposed  to  aerosolized  material  from 
the  waste  stream  or  to  the  waste  stream  itself.  The  investigators  examined  whether 
the  etiologic  agent  was  suspended  heavy  metals,  volatile  organic  compounds,  or 
intestinal  bacteria.  To  help  understand  the  exposures  and  identify  opportunities 
for  prevention,  the  investigators  interviewed  affected  employees  to  determine  if 
work  practices  contributed  to  employee  exposures. 

Over  several  months,  three  site  visits  were  made  to  the  Butler  County  landfill. 

HHE  investigators  observed  work  activities,  interviewed  five  employees  who 
worked  directly  with  the  land  application  process,  and  performed  exposure  moni¬ 
toring.  HHE  investigators  collected  area  and  personal  breathing  zone  air  samples 
for  culturable  bacteria,  endotoxin  from  Gram-negative  bacteria,  volatile  organic 
compounds,  and  trace  metals.  Bulk  samples  of  sewage  sludge  were  also  collected 
and  analyzed  to  determine  the  extent  of  contamination  with  coliform  bacteria. 

HHE  investigators  learned  that  the  employees  had  each  experienced  at  least  one 
episode  of  gastrointestinal  illness  soon  after  working  with  the  biosolids.  Employ¬ 
ees  reported  repeated  intermittent  episodes  of  gastrointestinal  symptoms  including 
diarrhea  and  abdominal  cramping.  The  detection  of  enteric  bacteria  in  the  air  and 
bulk  samples  collected  during  the  evaluation  indicated  a  potential  for  occupational 
exposure  to  disease-causing  microorganisms.  While  the  specific  component^)  of 
the  sewage  sludge  responsible  for  employees’  symptoms  were  not  determined,  the 
nature  and  timing  of  the  symptoms  suggested  occupational  exposure  by  ingestion 
or  inhalation  of  the  sludge  as  a  probable  cause. 

In  response  to  the  two  other  requests,  HHE  investigators  conducted  field  investi¬ 
gations  at  two  facilities  to  evaluate  worker  exposures  to  culturable  bacteria,  endo¬ 
toxin,  and  volatile  organic  compounds.  The  investigation  involving  strip  mine 
reclamation  activities  at  Power  Operating,  Inc.  in  Pennsylvania  was  completed  fol¬ 
lowing  an  initial  site  visit  and  medical  interviews.  Symptoms  of  nausea  and  upper 
respiratory  irritation  had  been  reported  after  a  biosolids  application.  No  additional 
biosolids  applications  occurred  after  the  HHE  request  was  received,  precluding 
opportunities  to  conduct  additional  assessments.  The  HHE  at  the  Outer  Loop 
Landfill  in  Kentucky  involved  area  and  personal  breathing  zone  monitoring  for 
culturable  bacteria  and  endotoxin  during  dumping  and  spreading  of  biosolids  at 
the  working  face  of  the  landfill.  Environmental  monitoring  data  showed  exposure 
to  culturable  enteric  organisms  and  endotoxin.  (NIOSH,  undated[c]) 

This  discussion  is  notable  for  its  skill  in  narrowing  the  issue  down  to  a  specific 
research  topic — exposure  to  aerosolized  waste  material — a  clear  discussion  of  the 
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activities  involved  in  studying  this  topic,  and  the  methodologies  that  guided  these 
activities. 

Outputs  and  Transfer:  What  were  the  major  outputs  from  this  research  area? 
How  and  to  whom  were  the  products  transferred?  This  section  highlights  the  rel¬ 
evance  of  outputs  and  transfer  activities.  The  discussion  should  be  precise  in  describ¬ 
ing  outputs  and  provide  sufficient  context  for  understanding  significance  of  outputs.  It 
should  avoid  bulleted  lists  of  outputs  and  transfers,  moving  detailed  citation  informa¬ 
tion  to  a  supporting-evidence  section.  Where  appropriate,  the  section  should  indicate 
customer  demand  for  outputs  and  emphasize  the  proactive  role  of  the  research  program 
in  transferring  program-generated  information  to  intermediate  and  final  customers. 
Both  formal  and  informal  venues  for  transferring  information  should  be  highlighted. 
Again,  the  HHE  example  is  useful: 

Outputs  and  Transfer.  HHE  investigators  produced  two  numbered  HHE  reports 
[Burton  and  Trout,  1999;  Delaney,  2003]  and  an  HHE  letter  report.  The  reports 
included  recommendations  to  improve  personal  hygiene,  modify  work  practices, 
and  use  personal  protective  equipment  to  minimize  exposure  to  biosolids,  and 
enhance  employee  training  regarding  these  issues.  HHE  investigators  contributed 
to  the  development  of  a  NIOSH  guidance  document  on  controlling  potential  occu¬ 
pational  [hazards]  related  to  exposure  to  Class  B  biosolids  (defined  as  those  that  are 
treated  but  still  contain  detectible  levels  of  pathogens)  [NIOSH,  2000].  The  docu¬ 
ment  was  revised  in  2002,  but  the  recommendations,  many  of  which  were  derived 
from  the  initial  HHE  report,  remained  unchanged  [NIOSH,  2002].  The  original 
NIOSH  document  was  republished  in  the  Journal  of  Applied  Occupational  and 
Environmental  Hygiene  [“Workers  Exposed  to  Class  B  Biosolids  During  and  After 
Field  Application,”  2001]. 

The  principal  HHE  investigator  was  invited  to  make  presentations  at  an  EPA 
meeting,  a  meeting  of  the  Kentucky-Tennessee  Water  Environment  Association, 
and  the  American  Industrial  Hygiene  Conference  and  Exposition.  To  ensure  that 
worker  safety  and  health  issues  were  being  adequately  addressed,  EPA  also  invited 
the  principal  HHE  investigator  to  serve  on  the  Pathogen  Equivalency  Committee, 
which  reviews  applications  for  new  waste  treatment  processes. 

Amidst  growing  concern  and  allegations  of  potential  health  hazards  from  biosolids 
applications,  and  following  a  hearing  by  the  U.S.  House  of  Representatives  Sci¬ 
ence  Committee  on  municipal  waste  disposal,  EPA  requested  that  the  National 
Research  Council  (NRC)  of  the  National  Academy  of  Sciences  review  the  science 
and  epidemiology  surrounding  biosolids.  Because  of  the  paucity  of  information 
on  worker  exposures  to  biosolids,  the  principal  HHE  investigator  was  invited  to 
an  NRC  meeting  to  summarize  results  of  the  NIOSH  HHEs.  In  2002,  the  NRC 
published  a  report  on  [its]  assessment,  citing  the  NIOSH  HHE  in  a  chapter  on  the 
epidemiologic  evidence  of  health  effects  associated  with  biosolids  production  and 
application  [NRC,  2002],  (NIOSH,  undated[c]) 
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This  discussion  has  a  number  of  strengths:  (1)  It  clearly  enumerates  the  research 
outputs  produced  by  HHE’s  activities;  (2)  it  describes  specific  transfer  activities  under¬ 
taken  to  ensure  that  the  research  outputs  found  their  way  to  users  who  needed  the 
information  (the  EPA,  the  U.S.  Congress,  state-level  organizations,  and  others)  and 
(3)  it  indicates  the  pull  aspect  of  the  transfer  activities  and  the  demand  on  the  part  of 
the  intermediate  customer  for  information.  It  also  provides  context  for  the  significance 
of  these  intermediate  outcomes  by  highlighting  the  paucity  of  existing  information  on 
the  subject,  heightening  the  need  for  and  interest  in  the  program’s  outputs. 

Intermediate  (or  End)  Outcomes:  What  effect  did  the  outputs  have  on  the 
broader  community?  This  section  emphasizes  the  effect  of  program  outputs.  It  estab¬ 
lishes  a  causal  thread  by  describing  intermediate  and  final  customers’  responses  to 
program  outputs.  The  discussion  plausibly  links  to  some  output.  If  there  are  multiple 
outcomes,  the  discussion  should  categorize  the  major  areas  of  outcomes.  Depending 
on  the  maturity  of  the  research  program,  it  may  not  have  identifiable  end  outcomes.  In 
these  cases,  it  is  appropriate  to  focus  on  the  intermediate  outcomes  that  have  derived 
from  the  program  activities.  However,  in  cases  in  which  intermediate  outcomes  appear 
to  have  led  to  end  outcomes,  including  a  section  (such  as  Progress  Toward  End  Out¬ 
comes)  is  important  to  emphasize  the  improved  societal  condition  that  can  be  linked 
to  the  research  program. 

The  HHE  example  for  this  subsection  is  effective  in  linking  the  intermediate 
outcomes  to  the  outputs  while  providing  specific  examples  of  customers  and  how  they 
used  the  information  (for  example,  to  develop  a  new  training  program  that  incorpo¬ 
rated  hazard  and  hygiene  recommendations): 

Intermediate  Outcomes.  The  Ohio  waste  treatment  facility  used  results  from  the 
HHE  to  develop  a  new  training  program  for  employees,  incorporating  the  hazard 
and  hygiene  recommendations  from  the  HHE  report. 

Trade  associations  and  other  interested  groups  have  disseminated  information  from 
these  HHE  reports  to  their  members.  Among  the  known  organizations  and  media 
outlets  that  have  disseminated  this  information  are  American  City  &  County  mag¬ 
azine;  Occupational  Hazards  magazine;  an  occupational  safety  and  health  listserv 
hosted  by  Duke  University;  The  Alliance  For  A  Clean  Environment,  a  non-profit 
environmental  advocacy  group;  AGnet,  a  food  safety  network  of  the  University  of 
Guelph,  Canada;  and  AgHealth  News,  a  newsletter  of  the  Agricultural  Health  and 
Safety  Center  of  the  University  of  California,  Davis.  (NIOSH,  undatedfc]) 

What's  Ahead:  What  are  some  specific  research  activities  currently  under  way 
or  in  planning  to  undertake  in  response  to  the  problem?  This  is  an  optional  section 
that  showcases  activities  or  outputs  currently  in  progress  or  in  the  planning  stages.  It 
should  include  work  that  has  not  yet  achieved  intermediate-outcome  status  but  with 
such  status  clearly  on  the  horizon  (three  to  five  years  out).  The  section  should  draw  on 
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the  logic  model  blueprint  to  discuss  how  this  work  will  lead  to  achievement  of  end  out¬ 
comes.  It  should  avoid  a  general  or  vague  description  of  what  is  needed  in  the  research 
held. 

HHE  research  on  chlorinated  compounds  included  the  following  discussion  in 
its  What’s  Ahead  section: 

The  HHE  Program  is  continuing  to  investigate  respiratory  symptoms,  skin  rash, 
and  eye  irritation  among  indoor  water  park  employees  at  the  Ohio  facility,  and  is 
assisting  the  NIOSH  Exposure  Assessment  [Coordinated]  Emphasis  Area  in  field 
testing  modifications  to  the  new  air  sampling  method  for  chloramines.  (NIOSH, 
undated[c]) 

As  the  example  suggests,  the  discussion  can  be  concise  and  straightforward.  Even 
after  the  program  achieved  some  important  end  outcomes,  it  continued  to  conduct 
work  in  this  area  to  investigate  ongoing  problems  and  improve  testing  methods. 

External  factors:  What  are  some  actions  or  forces  beyond  the  research  institu¬ 
tion's  control  that  have  important  bearing  on  potential  achievement  of  outcomes? 
In  discussing  the  contribution  of  research  to  outcomes,  it  is  important  to  identify  exter¬ 
nal  actions  or  forces,  beyond  the  control  of  the  research  institution,  that  bear  on  the 
translation  of  outputs  into  outcomes.  In  NIOSH’s  case,  industry,  labor,  regulators, 
or  other  entities  may  influence  the  ability  of  NIOSH  outputs  to  enhance  health  and 
safety.  This  section  may  be  a  stand-alone  one  or  be  included  with  each  section.  The  dis¬ 
cussion  should  clearly  articulate  how  the  external  factors  have  hindered  or  helped  the 
achievement  of  outcomes  (e.g.,  affected  activities,  production  of  outputs,  or  transfer). 
This  discussion  raises  a  potentially  delicate  issue:  It  should  convey  awareness  of  exter¬ 
nal  factors  without  using  them  as  an  excuse  or  justification  for  not  achieving  goals  or 
influencing  outcomes.  The  section  should  ideally  also  include  discussion  of  how  the 
research  program  is  responding  or  planning  to  respond  to  these  external  factors. 

The  HHE  examples  suggest  how  external  factors  played  into  the  outcomes  of 
its  biohazard  research  and  explains  how,  in  this  case,  external  factors  added  to  public 
awareness  of  the  problem,  increasing  interest  in  using  research  to  inform  a  solution: 

External  Factors.  The  extensive  media  coverage  surrounding  allegations  of  health 
problems  and  a  death  related  to  biosolids  application  served  to  focus  attention 
on  this  as  both  an  important  occupational  and  public  health  issue.  (NIOSH, 
undated[c]) 
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The  tools  discussed  in  this  book  are  commonly  used  in  evaluation  research.  The  use 
of  logic  models  to  articulate  program  theory  is  fundamental  to  many  types  of  evalu¬ 
ation.  Outcome  worksheets  are  similar  to  many  structured  frameworks  that  facilitate 
data  organization  for  analytical  purposes.  Outcome  narratives  draw  heavily  on  impact 
statements  as  a  way  to  convey  the  societal  benefits  of  research  to  a  broad  audience  and 
on  the  case-study  approach  as  a  way  to  tell  a  program’s  research  story.  In  this  context, 
the  utility  of  the  tools  discussed  in  this  book  lies  in  their  collective  use,  which  results  in 
linking  and  aligning  program  theory  to  data  collection  and  research  communication. 
Figure  6.1  provides  a  schematic  illustration  of  how  these  tools  are  linked. 

As  shown  in  Figure  6.1,  the  logic  model  served  as  the  foundation  for  all  the  other 
tools.  Methodologies  for  evaluating  research  impact  require  detailed  models  that  artic¬ 
ulate  the  program  theory  of  how  research  activities  contribute  to  outcomes.  The  logic 
model  provided  an  overarching  view  of  each  of  the  research  programs  and  served  as  the 
template  for  the  outcome  worksheets.  The  outcome  worksheets  were  instrumental  in 
identifying  the  specific  data  needed  to  show  cases  in  which  NIOSFI  research  contrib¬ 
uted  to  identifiable  outcomes.  The  outcome  narratives  provided  an  organizational  tem¬ 
plate  that  assisted  programs  in  communicating  their  impact  succinctly.  Drawn  from 
the  outcome  worksheets,  the  narratives  presented  the  outcome  evidence  in  a  relatively 
accessible  and  concise  format. 


Limitations  of  the  Tools 

Despite  their  usefulness  in  preparing  NIOS1T  programs  for  external  review,  these  tools 
had  limitations.  Representing  the  role  of  external  factors  in  the  logic  model  pathways 
was  one  area  of  difficulty.  External  factors  are  included  in  the  logic  models  as  a  line 
with  arrows  pointing  upward  beneath  the  logic  model  boxes.  Flowever,  more  atten¬ 
tion  could  have  been  given  to  identifying  some  of  the  external  factors  and  their  influ¬ 
ence  in  helping  or  hindering  the  achievement  of  outcomes.  Better  articulation  in  the 
logic  models  may  have  been  helpful  to  display,  where  known,  the  relative  magnitude 
of  these  factors.  For  example,  in  some  NIOSFI  programs,  the  inability  to  control  the 
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Figure  6.1 

Linking  Tools  for  Demonstrating  and  Communicating  Impact 
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budgetary  allocation  for  research  activities  played  a  pivotal  role  in  achieving  outcomes. 
Not  accounting  for  this  in  the  logic  model  may  have  created  a  false  sense  of  what  the 
program  could  control. 

External  factors  were  not  included  in  the  outcome  worksheet.  We  made  initial 
attempts  to  add  an  “External  Factors”  cell  within  each  of  the  columns  to  give  programs 
an  opportunity  to  describe  how  these  factors  affected  each  logic  model  element.  For 
many  research  programs,  the  effect  of  an  external  factor  was  not  limited  solely  to  an 
input,  activity,  or  output  but  had  varying  levels  of  influence  throughout  the  path  to 
outcomes.  Articulating  these  diffuse  effects  in  a  spreadsheet  format  was  not  easy.  Exter¬ 
nal  factors  were  discussed  in  the  outcome  narratives.  However,  given  the  importance 
of  external  factors  in  assessing  a  research  program’s  contribution  to  outcomes,  not  inte¬ 
grating  external  factors  into  all  three  of  the  tools  may  have  limited  their  applicability. 

The  format  for  the  outcome  worksheets  also  presented  a  number  of  challenges.  The 
spreadsheet  format,  though  helpful  for  organizing  and  aligning  information  in  a  single 
row  or  column,  was  not  the  best  layout  for  displaying  multiple  rows  and  columns.  This 
was  particularly  problematic  when  the  programs  tried  to  use  the  worksheets  to  review 
and  summarize  the  outcomes  associated  with  a  particular  strategic  goal.  Some  of  the 
programs  found  it  difficult  to  work  with  the  small  cells  of  the  spreadsheet.  The  cells 
were  used  to  promote  concise  and  succinct  explanations;  however,  inserting  text  and 
adjusting  for  the  small  type  was  a  hindrance  in  some  cases. 

In  addition  to  the  formatting  issues,  some  of  the  programs  found  it  difficult  to 
trace  backward  from  outcomes.  In  some  cases,  the  degree  of  overlap  among  research 
activities  made  it  difficult  to  distinguish  the  contribution  of  one  set  of  activities  from 
that  of  others.  In  other  cases,  programs  had  difficulty  prioritizing  the  most-significant 
outcomes  that  emerged  from  their  research  activities  within  the  designated  time  frame 
of  review. 

The  outcome-narrative  template  was  straightforward  in  its  application.  However, 
it  was  difficult  to  use  for  research  programs  that  had  not  yet  achieved  outcomes.  In  some 
instances,  we  recommended  using  headings,  such  as  “Progress  Toward  Outcomes”  to 
describe  outputs  that  had  gotten  as  far  as  transfer  but  did  not  have  identifiable  cus¬ 
tomer  use  or  application  associated  with  them.  The  appropriate  number  of  pages  for 
the  outcome  narratives  was  also  difficult  to  determine.  On  average,  we  advised  five  to 
seven  pages,  but  many  programs  exceeded  that  limit,  undermining  the  concision  and 
succinctness  objective  we  recommended  for  the  programs. 

Overall,  none  of  the  tools  reflected  the  inherent  complexities  of  research;  in  par¬ 
ticular,  they  did  not  reflect  the  feedback  loops  that  occur  throughout  the  process.  As 
discussed  in  Chapter  Three,  some  simplification  was  necessary  for  communication 
purposes.  However,  it  may  have  been  useful  to  provide  multiple  logic  models  of  the 
research  program  with  varying  levels  of  detail  and  allow  the  reviewers  to  choose  the  one 
most  appropriate  for  their  needs  in  assessing  program  impact. 
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Finally,  many  of  the  evidence  packages  themselves  tended  to  be  rather  dense.  The 
page  count  for  the  evidence  package  ranged  from  as  high  as  948  pages  to  as  low  as 
99.  Although  some  variation  in  the  sizes  of  the  evidence  packages  was  expected,  this 
extreme  range  may  indicate  that  the  standard  format  was  not  well-suited  for  all  of  the 
programs.  Moreover,  in  most  of  the  reviews,  the  ECs  still  requested  additional  informa¬ 
tion  from  the  research  programs.  In  some  cases,  the  information  requested  was  already 
included  in  their  evidence  packages  but  the  organizational  structure  and  the  density  of 
information  in  the  packages  made  it  difficult  to  access.  Thus,  providing  better  guide- 
posts  for  Ending  information  in  the  evidence  package  may  have  been  helpful. 


Other  Uses  for  the  Tools 

In  the  United  States  and  abroad,  program-evaluation  activities  are  becoming  more 
integrated  into  overall  systems  of  management  oriented  toward  results  (Cozzens,  2002). 
Thus,  the  tools  described  in  this  book  have  uses  beyond  supporting  preparations  for 
external  review.  For  example,  logic  models  can  directly  feed  into  project  planning  and 
outcome  monitoring  by  program  managers.  Outcome  monitoring  can  assist  program 
managers  in  thinking  through  the  data  they  will  need  to  demonstrate  the  extent  to 
which  their  program  or  even  a  specific  body  of  research  within  the  program  is  making 
a  difference  (Mayne,  1999).  The  logic  model  template  shown  in  Figure  6.2  provides  a 
structure  for  determining  whether  existing  strategic,  intermediate,  and  annual  goals 
are  aligned  with  program  operations.  These  goals  can  drive  the  development  of  mea¬ 
sures  (shown  in  red  in  Figure  6.2)  that  can  gauge  the  program’s  progress  toward  achiev¬ 
ing  outcomes. 

In  working  with  other  agencies,  we  have  used  logic  models  for  the  development 
of  a  portfolio  of  measures  that  align  with  goals  and  derive  from  the  logic  model  ele¬ 
ments.  Collectively,  these  measures  provide  a  means  of  assessing  (1)  the  use  of  resources 
to  achieve  management  objectives,  (2)  the  extent  to  which  activities  are  implemented 
as  expected,  (3)  the  number  and  quality  of  products  that  are  generated  from  research 
activities,  (4)  the  number  and  quality  of  transfer  activities  to  intermediate  custom¬ 
ers,  and  (5)  the  number  and  type  of  outcomes  based  on  the  customer  response  to  the 
outputs. 

In  addition,  measures  derived  from  the  logic  model  template  can  be  combined 
with  portfolio-management  tools  to  enhance  the  benefits  of  outcome  monitoring.  Port¬ 
folio  management  refers  to  a  class  of  methods  used  for  balancing  the  potential  value 
and  risk  of  individual  programs  within  a  portfolio  against  an  explicitly  defined  set 
of  goals.  There  are  several  methods  described  in  the  literature  for  conducting  portfo¬ 
lio  management,  including  the  balanced  scorecard  (Nair,  2004),  applied  information 
economics  (Flubbard,  2007),  IBM’s  Rational®  method  (Fianford,  2006),  earned-value 
management  (Fleming  and  Koppelman,  2005),  and  RAND’s  portfolio  management 
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Figure  6.2 

Logic  Model  Template  as  a  Program  Management  Tool 
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(PortMan)  and  assessment  method  (Silberglitt  et  ah,  2004).  Goals  and  measures  devel¬ 
oped  from  the  logic  model  can  be  used  for  evaluating  the  current  portfolio  of  R&D 
activities.  Similarly,  adjustments  to  the  current  R&D  portfolio  based  on  the  results  of 
portfolio  management  can  be  tracked  and  monitored  using  outcome  analysis  to  assess 
the  effect  of  these  adjustments  in  specific  societal  outcomes.  However,  there  may  be 
a  significant  delay  between  when  changes  in  a  given  portfolio  are  made  and  when 
changes  in  societal  outcomes  can  be  measured.  One  approach  to  improve  the  coupling 
between  the  two  methods  would  be  to  use  all  relevant  measures  associated  with  the 
annual,  intermediate,  and  strategic  goals.  Annual  measures  would  provide  information 
about  impact  of  the  most-recent  changes  to  the  portfolio  on  program  outputs,  while 
intermediate  and  strategic  measures  could  be  used  to  assess  the  impact  of  changes  on 
previous  portfolios.  It  would  also  be  important  that  changes  to  the  portfolio  be  exe¬ 
cuted  incrementally  in  order  to  assess  the  actual  impact  on  societal  outcomes.  None¬ 
theless,  combining  portfolio  management  with  outcome  analysis  has  the  potential  to 
provide  a  valuable  capability  to  those  seeking  not  only  to  achieve  the  highest  potential 
near-term  value  for  their  current  portfolio  but  also  to  ensure  that  they  are  achieving  the 
maximum  societal  benefit. 

The  outcome  worksheets  are  also  important  tools  that  could  be  used  for  outcome 
monitoring.  The  spreadsheet  format  enables  research  programs  to  look  across  the  data 
in  the  table  and  identify  which  set  of  research  activities  have  been  linked  to  outcomes, 
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assess  the  extent  to  which  transfer  activities  have  led  to  intended  customer  outcomes, 
and  identify  the  range  of  intended  and  unintended  outcomes  from  their  outputs.  Over 
time,  these  spreadsheets  can  become  the  foundation  of  a  database  that  tracks  uptake, 
adoption,  and  utility  of  research  outputs  by  different  customers.  This  could  enable 
better  strategic  planning  of  transfer  activities  and  working  more  effectively  with  part¬ 
ners  at  early  project  phases.  As  programs  consider  the  causal  linkages,  they  may  realize 
that  more  data  may  need  to  be  collected  on  factors  that  influence  customer  adoption 
of  technologies  rather  than  collecting  data  on  outcome  measures  of  customer  adoption 
and  use  of  certain  technologies. 

Evidence  packages  and  outcome  narratives  are  useful  tools  for  communicating 
impact  to  audiences  beyond  just  external  reviewers.  An  effective  outcome  narrative 
can  communicate  to  decisionmakers  not  only  the  impact  of  research  programs  but  also 
their  value  to  stakeholders,  such  as  legislators,  local  officials,  and  the  general  public. 
These  stakeholders  prefer  information  that  gets  to  the  point  quickly  and  clearly.  The 
concise  format  and  readable  layout  of  the  outcome  narrative  ensures  that  impacts  can 
be  accessed  and  appreciated  by  a  broad  community  of  stakeholders. 

Clearly,  more  tools  and  methods  are  needed  to  map  the  causal  connections 
between  publicly  funded  research  and  its  social  benefits.  However,  putting  the  tools 
described  here  to  rigorous  use  is  an  important  step  in  determining  whether  federally 
funded  research  programs  are  achieving  long-range  societal  goals. 
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NIOSH  Program  Logic  Model 


This  appendix  presents  the  NIOSH  program  logic  model. 
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Figure  A.1 

NIOSH  Program  Logic  Model 
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SOURCE:  National  Academies  (2007). 

NOTE:  ARC  =  agricultural  research  center.  WHO  =  World  Health  Organization. 

3lntramural  and  extramural,  including  domestic  and  international  efforts,  such  as  work  conducted  at  ERCs,  ARCs,  and  Global  Network  of  the  WHO 
Network  of  Collaborating  Centres. 
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